http://wiki.apache.org/solr/WritingDistributedSearchComponents

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

http://wiki.apache.org/solr/WritingDistributedSearchComponents

Benson Margulies
I'd like to add some more information to this; to that end, I'd
appreciate a correctness review of the following:

* QueryComponent distribution algorithm

The query component distributes queries in two passes. First, it sends
the query out to the shards with the field list trimmed to be just the
score and the doc ID. Upon receipt of these results, it merges the
lists, and then issues new queries out to the shards with
ids=id1,id2,... to retrieve the remaining fields. The results are
packaged up and flow out through the result writer.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: http://wiki.apache.org/solr/WritingDistributedSearchComponents

Yonik Seeley-2-2
On Wed, Apr 11, 2012 at 8:15 PM, Benson Margulies <[hidden email]> wrote:

> I'd like to add some more information to this; to that end, I'd
> appreciate a correctness review of the following:
>
> * QueryComponent distribution algorithm
>
> The query component distributes queries in two passes. First, it sends
> the query out to the shards with the field list trimmed to be just the
> score and the doc ID. Upon receipt of these results, it merges the
> lists, and then issues new queries out to the shards with
> ids=id1,id2,... to retrieve the remaining fields. The results are
> packaged up and flow out through the result writer.

Yep.

-Yonik
lucenerevolution.com - Lucene/Solr Open Source Search Conference.
Boston May 7-10

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: http://wiki.apache.org/solr/WritingDistributedSearchComponents

Mikhail Khludnev
Btw, I always wanted to ask why it's always done in two stages. It seems to me that it's purposed for the specific usecase. But shouldn't we have an option to do it in the single stage? eg. if I have a few fields to retrieve and not really deep paging, I'd like to retrieve whole fl-s and merge them by PKs. 

Benson,
Thanks for contributing the doc for community. I spend a lot of time to get into that machinery on my own.  

Regards

On Thu, Apr 12, 2012 at 4:17 AM, Yonik Seeley <[hidden email]> wrote:
On Wed, Apr 11, 2012 at 8:15 PM, Benson Margulies <[hidden email]> wrote:
> I'd like to add some more information to this; to that end, I'd
> appreciate a correctness review of the following:
>
> * QueryComponent distribution algorithm
>
> The query component distributes queries in two passes. First, it sends
> the query out to the shards with the field list trimmed to be just the
> score and the doc ID. Upon receipt of these results, it merges the
> lists, and then issues new queries out to the shards with
> ids=id1,id2,... to retrieve the remaining fields. The results are
> packaged up and flow out through the result writer.

Yep.

-Yonik
lucenerevolution.com - Lucene/Solr Open Source Search Conference.
Boston May 7-10

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]




--
Sincerely yours
Mikhail Khludnev

Reply | Threaded
Open this post in threaded view
|

Re: http://wiki.apache.org/solr/WritingDistributedSearchComponents

Li Li
me too. maybe it should provide a one-stage component.

On Fri, Apr 13, 2012 at 1:41 AM, Mikhail Khludnev <[hidden email]> wrote:
Btw, I always wanted to ask why it's always done in two stages. It seems to me that it's purposed for the specific usecase. But shouldn't we have an option to do it in the single stage? eg. if I have a few fields to retrieve and not really deep paging, I'd like to retrieve whole fl-s and merge them by PKs. 

Benson,
Thanks for contributing the doc for community. I spend a lot of time to get into that machinery on my own.  

Regards


On Thu, Apr 12, 2012 at 4:17 AM, Yonik Seeley <[hidden email]> wrote:
On Wed, Apr 11, 2012 at 8:15 PM, Benson Margulies <[hidden email]> wrote:
> I'd like to add some more information to this; to that end, I'd
> appreciate a correctness review of the following:
>
> * QueryComponent distribution algorithm
>
> The query component distributes queries in two passes. First, it sends
> the query out to the shards with the field list trimmed to be just the
> score and the doc ID. Upon receipt of these results, it merges the
> lists, and then issues new queries out to the shards with
> ids=id1,id2,... to retrieve the remaining fields. The results are
> packaged up and flow out through the result writer.

Yep.

-Yonik
lucenerevolution.com - Lucene/Solr Open Source Search Conference.
Boston May 7-10

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]




--
Sincerely yours
Mikhail Khludnev