Copying the request parameters to Solr's response

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Copying the request parameters to Solr's response

Bertrand Delacretaz
Hi,

I need to implement paging of Solr result sets, and (unless I have
overlooked something that already exists) it would be useful to copy
the request parameters to the output.

I'm thinking of adding something like this to the XML output:

  <responseHeader>
      <lst name="queryParameters">
        <str name="q">author:Leonardo</str>
        <str name="start">24</str>
        <str name="rows">12</str>
   etc...

I don't think the SolrParams class provides an Iterator to retrieve
all parameters, I'll add one to implement this.

WDYT?

-Bertrand
Reply | Threaded
Open this post in threaded view
|

Re: Copying the request parameters to Solr's response

Erik Hatcher
I think its a good idea, but it probably should be made optional.  
Clients can keep track of the state themselves, and keeping the  
response size as small as possible is valuable.  But it would be  
helpful in some situations for the client to get the original query  
context sent back too.

        Erik


On Oct 24, 2006, at 4:20 AM, Bertrand Delacretaz wrote:

> Hi,
>
> I need to implement paging of Solr result sets, and (unless I have
> overlooked something that already exists) it would be useful to copy
> the request parameters to the output.
>
> I'm thinking of adding something like this to the XML output:
>
>  <responseHeader>
>      <lst name="queryParameters">
>        <str name="q">author:Leonardo</str>
>        <str name="start">24</str>
>        <str name="rows">12</str>
>   etc...
>
> I don't think the SolrParams class provides an Iterator to retrieve
> all parameters, I'll add one to implement this.
>
> WDYT?
>
> -Bertrand

Reply | Threaded
Open this post in threaded view
|

Re: Copying the request parameters to Solr's response

Bertrand Delacretaz
On 10/24/06, Erik Hatcher <[hidden email]> wrote:
> I think its a good idea, but it probably should be made optional...

Yes - I was meaning to make it optional, forgot to mention that.

-Bertrand
Reply | Threaded
Open this post in threaded view
|

Re: Copying the request parameters to Solr's response

Yonik Seeley-2
In reply to this post by Bertrand Delacretaz
On 10/24/06, Bertrand Delacretaz <[hidden email]> wrote:

> I need to implement paging of Solr result sets, and (unless I have
> overlooked something that already exists) it would be useful to copy
> the request parameters to the output.
>
> I'm thinking of adding something like this to the XML output:
>
>   <responseHeader>
>       <lst name="queryParameters">
>         <str name="q">author:Leonardo</str>
>         <str name="start">24</str>
>         <str name="rows">12</str>
>    etc...

+1
I had been thinking of doing that to allow for more stateless clients
that don't even know what it was they queried for.

I imagine this would just be for explicitly passed parameters?

> I don't think the SolrParams class provides an Iterator to retrieve
> all parameters, I'll add one to implement this.

Definitely.  It also hit me just yesterday that this was missing while
I was thinking about how to solve something else.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Copying the request parameters to Solr's response

Yonik Seeley-2
In reply to this post by Bertrand Delacretaz
On 10/24/06, Bertrand Delacretaz <[hidden email]> wrote:
>   <responseHeader>
>       <lst name="queryParameters">
>         <str name="q">author:Leonardo</str>
>         <str name="start">24</str>
>         <str name="rows">12</str>

What do people think... should this (optionally) go under
responseHeader, to be added by the ResponseWriter when it sees the
flag to do so,
  OR
Should it be added in the response body, at the same level as
highlighting info, faceting info, etc.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Copying the request parameters to Solr's response

Bertrand Delacretaz
In reply to this post by Yonik Seeley-2
On 10/24/06, Yonik Seeley <[hidden email]> wrote:

> ...I imagine this would just be for explicitly passed parameters?...

I think so, the defaults would be re-applied anyway, if the client
makes another request with the same parameters.

-Bertrand
Reply | Threaded
Open this post in threaded view
|

Re: Copying the request parameters to Solr's response

Walter Underwood, Netflix
In reply to this post by Erik Hatcher
Returning the query parameters is really useful. I'm not sure it
needs to be optional, they are small and options multiply the test
cases.

It can even be useful to return the values of the defaults.

All those go into the key for any client side caching, for example.

wunder

On 10/24/06 1:55 AM, "Erik Hatcher" <[hidden email]> wrote:

> I think its a good idea, but it probably should be made optional.
> Clients can keep track of the state themselves, and keeping the
> response size as small as possible is valuable.  But it would be
> helpful in some situations for the client to get the original query
> context sent back too.
>
> Erik
>
>
> On Oct 24, 2006, at 4:20 AM, Bertrand Delacretaz wrote:
>
>> Hi,
>>
>> I need to implement paging of Solr result sets, and (unless I have
>> overlooked something that already exists) it would be useful to copy
>> the request parameters to the output.
>>
>> I'm thinking of adding something like this to the XML output:
>>
>>  <responseHeader>
>>      <lst name="queryParameters">
>>        <str name="q">author:Leonardo</str>
>>        <str name="start">24</str>
>>        <str name="rows">12</str>
>>   etc...
>>
>> I don't think the SolrParams class provides an Iterator to retrieve
>> all parameters, I'll add one to implement this.
>>
>> WDYT?
>>
>> -Bertrand
>

Reply | Threaded
Open this post in threaded view
|

Re: Copying the request parameters to Solr's response

Walter Underwood, Netflix
In reply to this post by Bertrand Delacretaz
On 10/24/06 7:22 AM, "Bertrand Delacretaz" <[hidden email]> wrote:
> On 10/24/06, Yonik Seeley <[hidden email]> wrote:
>
>> ...I imagine this would just be for explicitly passed parameters?...
>
> I think so, the defaults would be re-applied anyway, if the client
> makes another request with the same parameters.
>
> -Bertrand

The defaults can change, especially if the client saves results.
If possible, you want to return a full context for the results.

Ultraseek has had XML results for several years and a full query
context would have been useful in several situations. The Ultraseek
result format targeted a different problem, returning enough info
to calculate a global IDF across multiple collections and re-score
the combined results.

http://search.ultraseek.com/saquery.xml?qt=saquery.xml&col=usdc&col=docs

The Java client library for Ultraseek (XPA) does keep a local results
cache and uses the query plus the query context as a key.

wunder
--
Walter Underwood
Search Guru, Netflix
Former Ultraseek Architect

Reply | Threaded
Open this post in threaded view
|

Re: Copying the request parameters to Solr's response

Bertrand Delacretaz
On 10/24/06, Walter Underwood <[hidden email]> wrote:

> ...The defaults can change, especially if the client saves results.
> If possible, you want to return a full context for the results...

It wouldn't cost much anyway, so I think you're right that this is useful.

For now, I have uploaded a first patch at
https://issues.apache.org/jira/browse/SOLR-59, feel free to play with
it or improve it. It doesn't include the defaults (and I have to run
now, will look at that tomorrow or Thursday).

-Bertrand
Reply | Threaded
Open this post in threaded view
|

Re: Re: Copying the request parameters to Solr's response

Mike Klaas
In reply to this post by Walter Underwood, Netflix
On 10/24/06, Walter Underwood <[hidden email]> wrote:

> The defaults can change, especially if the client saves results.
> If possible, you want to return a full context for the results.

Perhaps, but the defaults are an ever-expanding list of params and
usually much larger than the provided params.

echoParams={false/explicit/all}

-Mike
Reply | Threaded
Open this post in threaded view
|

Re: Re: Copying the request parameters to Solr's response

Yonik Seeley-2
On 10/24/06, Mike Klaas <[hidden email]> wrote:
> Perhaps, but the defaults are an ever-expanding list of params and
> usually much larger than the provided params.
>
> echoParams={false/explicit/all}

+1

"all" will be very useful for debugging too.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Copying the request parameters to Solr's response

Chris Hostetter-3
In reply to this post by Yonik Seeley-2
: What do people think... should this (optionally) go under
: responseHeader, to be added by the ResponseWriter when it sees the
: flag to do so,
:   OR
: Should it be added in the response body, at the same level as
: highlighting info, faceting info, etc.

I think it should be done by the RequestHandlers, using shared utility
methods to make it easy, and thus should go in the response body -- not
hte header.

the header should be kept small, containing only core low level info about
the status of the communication with SolrCore.


-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Copying the request parameters to Solr's response

Chris Hostetter-3

Acctually ... i revise my previous statment ... if we are talking about
just the explicit params that come from the client in the request (the
same that get written to the log) then i suppose letting SolrCore put them
in the response makes sense ... in which case putting them in the header
is right thing to do ... this seems like the most important use case
because it's what is neccessary for the client to recreate the given
request (someone made the point that defaults can change, that may be
true, but the code implimenting the request handler could change just as
easily -- nothing we do can 100% garuntee that the user gets hte same
behavior).

The other cases: logging the "effective" params, what the plugin
ultimately wound up using after default and invarient params might get
applied ... that's a complicated issue in my mind .. i think if people
wnat that functionality in their RequestHandlers that's up to them ... i
don't know if it's a good idea for standard or dismax to include them.


: I think it should be done by the RequestHandlers, using shared utility
: methods to make it easy, and thus should go in the response body -- not
: hte header.
:
: the header should be kept small, containing only core low level info about
: the status of the communication with SolrCore.


-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Copying the request parameters to Solr's response

Bill Au
+1

I think it should be optional, default to off, and the request parameters
should be in the header.  One advantage of having it in the header is that
this information can be used by the clinet to process the query responses as
they are being read.

Bill

On 10/24/06, Chris Hostetter <[hidden email]> wrote:

>
>
> Acctually ... i revise my previous statment ... if we are talking about
> just the explicit params that come from the client in the request (the
> same that get written to the log) then i suppose letting SolrCore put them
> in the response makes sense ... in which case putting them in the header
> is right thing to do ... this seems like the most important use case
> because it's what is neccessary for the client to recreate the given
> request (someone made the point that defaults can change, that may be
> true, but the code implimenting the request handler could change just as
> easily -- nothing we do can 100% garuntee that the user gets hte same
> behavior).
>
> The other cases: logging the "effective" params, what the plugin
> ultimately wound up using after default and invarient params might get
> applied ... that's a complicated issue in my mind .. i think if people
> wnat that functionality in their RequestHandlers that's up to them ... i
> don't know if it's a good idea for standard or dismax to include them.
>
>
> : I think it should be done by the RequestHandlers, using shared utility
> : methods to make it easy, and thus should go in the response body -- not
> : hte header.
> :
> : the header should be kept small, containing only core low level info
> about
> : the status of the communication with SolrCore.
>
>
> -Hoss
>
>