What is the best way in Solr to perform iterative search?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

What is the best way in Solr to perform iterative search?

karl.wright
Folks,
 
I have a search task which needs to do the following:
 
  • Do a search
  • Use the results of that search to form a final query, which when executed returns the overall results
 
The question is, what is the best way to build components within Solr to handle this request flow?  My first thought was to construct two QParserPlugin objects, one for each query, and then perhaps write a custom version of QueryComponent that does the job.   But I certainly can’t put all the above logic into one SearchComponent, because of the requirement that I need the results from one search to do the subsequent one.  I *could* replace or extend the requestHandler instead.  Or, I could have multiple SearchComponent’s in the pipeline where one looks at the other’s results (will this work???)  Any other ideas?
 
Karl
 
 
Reply | Threaded
Open this post in threaded view
|

Re: What is the best way in Solr to perform iterative search?

Erik Hatcher-4
I'm not quite clear on what you want ... if you want to "search within  
previous search results", simply take the original query and turn it  
into an fq parameter for the next request using the new query as the q  
parameter.

If you want to take some part of the current results to form a new  
query internally to Solr, it sounds like what you want is a custom  
QueryComponent.  I've done this, for example, to back off a search  
that had zero results and broaden it automatically.  Register a new  
QueryComponent as "query" and it'll replace the existing one - in  
solrconfig.xml.

        Erik


On Aug 3, 2010, at 12:18 PM, <[hidden email]> <[hidden email]
 > wrote:

> Folks,
>
> I have a search task which needs to do the following:
>
> • Do a search
> • Use the results of that search to form a final query, which when  
> executed returns the overall results
>
> The question is, what is the best way to build components within  
> Solr to handle this request flow?  My first thought was to construct  
> two QParserPlugin objects, one for each query, and then perhaps  
> write a custom version of QueryComponent that does the job.   But I  
> certainly can’t put all the above logic into one SearchComponent,  
> because of the requirement that I need the results from one search  
> to do the subsequent one.  I *could* replace or extend the  
> requestHandler instead.  Or, I could have multiple SearchComponent’s  
> in the pipeline where one looks at the other’s results (will this  
> work???)  Any other ideas?
>
> Karl
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: What is the best way in Solr to perform iterative search?

karl.wright
Thanks - I've got the latter situation, actually.

I presume when you wrote your QueryComponent you did the query first execution, modification, and second execution all within the equivalent of the prepare() method?  I didn't want to replace QueryComponent itself, since it does a fair bit of stuff, but maybe I can extend it and just override the appropriate method...

Karl


-----Original Message-----
From: ext Erik Hatcher [mailto:[hidden email]]
Sent: Tuesday, August 03, 2010 1:01 PM
To: [hidden email]
Subject: Re: What is the best way in Solr to perform iterative search?

I'm not quite clear on what you want ... if you want to "search within  
previous search results", simply take the original query and turn it  
into an fq parameter for the next request using the new query as the q  
parameter.

If you want to take some part of the current results to form a new  
query internally to Solr, it sounds like what you want is a custom  
QueryComponent.  I've done this, for example, to back off a search  
that had zero results and broaden it automatically.  Register a new  
QueryComponent as "query" and it'll replace the existing one - in  
solrconfig.xml.

        Erik


On Aug 3, 2010, at 12:18 PM, <[hidden email]> <[hidden email]
 > wrote:

> Folks,
>
> I have a search task which needs to do the following:
>
> * Do a search
> * Use the results of that search to form a final query, which when  
> executed returns the overall results
>
> The question is, what is the best way to build components within  
> Solr to handle this request flow?  My first thought was to construct  
> two QParserPlugin objects, one for each query, and then perhaps  
> write a custom version of QueryComponent that does the job.   But I  
> certainly can't put all the above logic into one SearchComponent,  
> because of the requirement that I need the results from one search  
> to do the subsequent one.  I *could* replace or extend the  
> requestHandler instead.  Or, I could have multiple SearchComponent's  
> in the pipeline where one looks at the other's results (will this  
> work???)  Any other ideas?
>
> Karl
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: What is the best way in Solr to perform iterative search?

karl.wright
Oops - I meant the "process()" method, not "prepare()", below.

Looking at the code, it occurs to me that ideally such sequences of queries ought to have their own supporting infrastructure within Solr, because doing two queries inside a search component's process() method basically defeats query distribution and other nice solr features.  The only thing that prevents you from doing such a thing at the SearchHandler level is the lack of ability for any given query in a sequence from seeing the results of any previous query(s) - there's simply no way to pass that info around at present.  So probably the request object would need an additional hash map or some such, to allow such information transfer, and the search handler configuration would need to include a named and ordered set of individual query component lists.  Does anyone have any interest in such a feature?

Karl


-----Original Message-----
From: Wright Karl (Nokia-MS/Cambridge)
Sent: Tuesday, August 03, 2010 1:16 PM
To: [hidden email]
Subject: RE: What is the best way in Solr to perform iterative search?

Thanks - I've got the latter situation, actually.

I presume when you wrote your QueryComponent you did the query first execution, modification, and second execution all within the equivalent of the prepare() method?  I didn't want to replace QueryComponent itself, since it does a fair bit of stuff, but maybe I can extend it and just override the appropriate method...

Karl


-----Original Message-----
From: ext Erik Hatcher [mailto:[hidden email]]
Sent: Tuesday, August 03, 2010 1:01 PM
To: [hidden email]
Subject: Re: What is the best way in Solr to perform iterative search?

I'm not quite clear on what you want ... if you want to "search within  
previous search results", simply take the original query and turn it  
into an fq parameter for the next request using the new query as the q  
parameter.

If you want to take some part of the current results to form a new  
query internally to Solr, it sounds like what you want is a custom  
QueryComponent.  I've done this, for example, to back off a search  
that had zero results and broaden it automatically.  Register a new  
QueryComponent as "query" and it'll replace the existing one - in  
solrconfig.xml.

        Erik


On Aug 3, 2010, at 12:18 PM, <[hidden email]> <[hidden email]
 > wrote:

> Folks,
>
> I have a search task which needs to do the following:
>
> * Do a search
> * Use the results of that search to form a final query, which when  
> executed returns the overall results
>
> The question is, what is the best way to build components within  
> Solr to handle this request flow?  My first thought was to construct  
> two QParserPlugin objects, one for each query, and then perhaps  
> write a custom version of QueryComponent that does the job.   But I  
> certainly can't put all the above logic into one SearchComponent,  
> because of the requirement that I need the results from one search  
> to do the subsequent one.  I *could* replace or extend the  
> requestHandler instead.  Or, I could have multiple SearchComponent's  
> in the pipeline where one looks at the other's results (will this  
> work???)  Any other ideas?
>
> Karl
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: What is the best way in Solr to perform iterative search?

Koji Sekiguchi
I have a similar requirement:

https://issues.apache.org/jira/browse/SOLR-1878

Koji

--
http://www.rondhuit.com/en/



(10/08/04 4:41), [hidden email] wrote:

> Oops - I meant the "process()" method, not "prepare()", below.
>
> Looking at the code, it occurs to me that ideally such sequences of queries ought to have their own supporting infrastructure within Solr, because doing two queries inside a search component's process() method basically defeats query distribution and other nice solr features.  The only thing that prevents you from doing such a thing at the SearchHandler level is the lack of ability for any given query in a sequence from seeing the results of any previous query(s) - there's simply no way to pass that info around at present.  So probably the request object would need an additional hash map or some such, to allow such information transfer, and the search handler configuration would need to include a named and ordered set of individual query component lists.  Does anyone have any interest in such a feature?
>
> Karl
>
>
> -----Original Message-----
> From: Wright Karl (Nokia-MS/Cambridge)
> Sent: Tuesday, August 03, 2010 1:16 PM
> To: [hidden email]
> Subject: RE: What is the best way in Solr to perform iterative search?
>
> Thanks - I've got the latter situation, actually.
>
> I presume when you wrote your QueryComponent you did the query first execution, modification, and second execution all within the equivalent of the prepare() method?  I didn't want to replace QueryComponent itself, since it does a fair bit of stuff, but maybe I can extend it and just override the appropriate method...
>
> Karl
>
>
> -----Original Message-----
> From: ext Erik Hatcher [mailto:[hidden email]]
> Sent: Tuesday, August 03, 2010 1:01 PM
> To: [hidden email]
> Subject: Re: What is the best way in Solr to perform iterative search?
>
> I'm not quite clear on what you want ... if you want to "search within
> previous search results", simply take the original query and turn it
> into an fq parameter for the next request using the new query as the q
> parameter.
>
> If you want to take some part of the current results to form a new
> query internally to Solr, it sounds like what you want is a custom
> QueryComponent.  I've done this, for example, to back off a search
> that had zero results and broaden it automatically.  Register a new
> QueryComponent as "query" and it'll replace the existing one - in
> solrconfig.xml.
>
> Erik
>
>
> On Aug 3, 2010, at 12:18 PM,<[hidden email]>  <[hidden email]
>   >  wrote:
>
>    
>> Folks,
>>
>> I have a search task which needs to do the following:
>>
>> * Do a search
>> * Use the results of that search to form a final query, which when
>> executed returns the overall results
>>
>> The question is, what is the best way to build components within
>> Solr to handle this request flow?  My first thought was to construct
>> two QParserPlugin objects, one for each query, and then perhaps
>> write a custom version of QueryComponent that does the job.   But I
>> certainly can't put all the above logic into one SearchComponent,
>> because of the requirement that I need the results from one search
>> to do the subsequent one.  I *could* replace or extend the
>> requestHandler instead.  Or, I could have multiple SearchComponent's
>> in the pipeline where one looks at the other's results (will this
>> work???)  Any other ideas?
>>
>> Karl
>>
>>
>>      
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
>    


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: What is the best way in Solr to perform iterative search?

karl.wright
I think a general feature along the lines I discussed would permit *all* of the scenarios to be dealt with effectively.
Very good - I'll create a feature, ticket, and patch, and link to your ticket as well. ;-)

Thanks,
Karl

________________________________________
From: ext Koji Sekiguchi [[hidden email]]
Sent: Tuesday, August 03, 2010 7:34 PM
To: [hidden email]
Subject: Re: What is the best way in Solr to perform iterative search?

I have a similar requirement:

https://issues.apache.org/jira/browse/SOLR-1878

Koji

--
http://www.rondhuit.com/en/



(10/08/04 4:41), [hidden email] wrote:

> Oops - I meant the "process()" method, not "prepare()", below.
>
> Looking at the code, it occurs to me that ideally such sequences of queries ought to have their own supporting infrastructure within Solr, because doing two queries inside a search component's process() method basically defeats query distribution and other nice solr features.  The only thing that prevents you from doing such a thing at the SearchHandler level is the lack of ability for any given query in a sequence from seeing the results of any previous query(s) - there's simply no way to pass that info around at present.  So probably the request object would need an additional hash map or some such, to allow such information transfer, and the search handler configuration would need to include a named and ordered set of individual query component lists.  Does anyone have any interest in such a feature?
>
> Karl
>
>
> -----Original Message-----
> From: Wright Karl (Nokia-MS/Cambridge)
> Sent: Tuesday, August 03, 2010 1:16 PM
> To: [hidden email]
> Subject: RE: What is the best way in Solr to perform iterative search?
>
> Thanks - I've got the latter situation, actually.
>
> I presume when you wrote your QueryComponent you did the query first execution, modification, and second execution all within the equivalent of the prepare() method?  I didn't want to replace QueryComponent itself, since it does a fair bit of stuff, but maybe I can extend it and just override the appropriate method...
>
> Karl
>
>
> -----Original Message-----
> From: ext Erik Hatcher [mailto:[hidden email]]
> Sent: Tuesday, August 03, 2010 1:01 PM
> To: [hidden email]
> Subject: Re: What is the best way in Solr to perform iterative search?
>
> I'm not quite clear on what you want ... if you want to "search within
> previous search results", simply take the original query and turn it
> into an fq parameter for the next request using the new query as the q
> parameter.
>
> If you want to take some part of the current results to form a new
> query internally to Solr, it sounds like what you want is a custom
> QueryComponent.  I've done this, for example, to back off a search
> that had zero results and broaden it automatically.  Register a new
> QueryComponent as "query" and it'll replace the existing one - in
> solrconfig.xml.
>
>       Erik
>
>
> On Aug 3, 2010, at 12:18 PM,<[hidden email]>  <[hidden email]
>   >  wrote:
>
>
>> Folks,
>>
>> I have a search task which needs to do the following:
>>
>>      * Do a search
>>      * Use the results of that search to form a final query, which when
>> executed returns the overall results
>>
>> The question is, what is the best way to build components within
>> Solr to handle this request flow?  My first thought was to construct
>> two QParserPlugin objects, one for each query, and then perhaps
>> write a custom version of QueryComponent that does the job.   But I
>> certainly can't put all the above logic into one SearchComponent,
>> because of the requirement that I need the results from one search
>> to do the subsequent one.  I *could* replace or extend the
>> requestHandler instead.  Or, I could have multiple SearchComponent's
>> in the pipeline where one looks at the other's results (will this
>> work???)  Any other ideas?
>>
>> Karl
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]