nested querries, and LocalParams syntax

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

nested querries, and LocalParams syntax

Jonathan Rochkind
So I'm trying to wrap my head around nested querries. Also that thing
that isn't a nested query, but is similar, which I think is called
"LocalParams syntax",  like:

&q={!dismax qf=$something}cat dog

(All my examples are not URL-encoded for clarity, of course they'd have
to be before sending to solr).

I've been referring to both:
http://wiki.apache.org/solr/SolrQuerySyntax
http://www.lucidimagination.com/blog/2009/03/31/nested-queries-in-solr/

Let's start with that not-nested query example.   Can you in fact use it
as above, to force dismax handling of the 'q' even if the qt or request
handler default is something else?  The documentation is confusing: "In
standard Solr search handlers, the default type of the main query only
may be specified via the defType parameter. The default type of all
other query parameters will remain "lucene
<http://wiki.apache.org/solr/SolrQuerySyntax#lucene>"."

I _think_ it's trying to say that I _can_, even in a standard search
handler, force dismax with {!dismax}, I just can't change the type of
_other_ query parameters -- rather than saying that I _can't_ use
{!dismax} to force dismax type of 'q' in a "standard search handler".  Yes?

Next, when we get to dismax---can I embed parameter value literals in
there, and how?

q={!dismax qf="field^2 field2^4"} some query

With quotes like that? What if a parameter value literal itself has a
quote mark in it, is there a way to escape it? (Say, can I embed an fq
in there whose value itself needs quotes?)

Will parameters supplied in LocalParams like that always "override"
something supplied in a straight query parameter?

&qf=somefield&q={!dismax qf=otherfield} cat dog
    => I'll get "otherfield" and not "somefield" ?

Can you use the LocalParams _without_ a !defType, to just supply
parameters but take the default defType for the request handler?    
&q={qf=somefield} cat dog

And now we move into the nested querries -- for the simplest case nested
query, is it in fact identical to using LocalParams without a nested query?

&q={!dismax qf=something} cat dog

===?

&q=_query_:"{!dismax qf=somefield} cat dog"

Are those equivelent?  In nested querries like that, the issue of quote
escaping becomes even more confusing, can I put a literal in there,
and... how?

&q=_query_:"{!dismax qf="somefield^1 otherfield^2"}"   ==> obviously
won't work, the quotes are interfering with each other.

Phew, that's a lot of questions, but probably just a start, heh. I'm
trying to wrap my head around the complete possibilities of LocalParams
and nested querries, to figure out if I can use them to do the kind of
complicated thing I'm trying to do.

I guess I could try to test all these things out and figure out the
answer, and probably will proceed to do that if nobody else has any
other advice, but I figured it didn't hurt to ask, and some of these
things will be hard to guess on my own (like the quote escaping issue).  
If I figure some stuff out, is there any good community place (wiki
somewhere?) for me to document this stuff for others?

Thanks for any advice,

Jonathan

Reply | Threaded
Open this post in threaded view
|

Re: nested querries, and LocalParams syntax

Yonik Seeley-2-2
Have you seen
http://wiki.apache.org/solr/LocalParams

It may answer some of the questions, such as stating that "backslash
escaping works within quoted strings".

I'd encourage you to try things out with the example server and adding
debugQuery=true to your requests... it's the easiest way to figure out
what's going on.

defType is just the default type in the event that the main query
doesn't have local params that specifically say what the type of the
query is.

As far as overriding, it's on a per QueryParserPlugin basis, but most
do use localParams to override gobal request parameters.

-Yonik
http://www.lucidimagination.com



On Wed, May 26, 2010 at 11:09 AM, Jonathan Rochkind <[hidden email]> wrote:

> So I'm trying to wrap my head around nested querries. Also that thing that
> isn't a nested query, but is similar, which I think is called "LocalParams
> syntax",  like:
>
> &q={!dismax qf=$something}cat dog
>
> (All my examples are not URL-encoded for clarity, of course they'd have to
> be before sending to solr).
>
> I've been referring to both:
> http://wiki.apache.org/solr/SolrQuerySyntax
> http://www.lucidimagination.com/blog/2009/03/31/nested-queries-in-solr/
>
> Let's start with that not-nested query example.   Can you in fact use it as
> above, to force dismax handling of the 'q' even if the qt or request handler
> default is something else?  The documentation is confusing: "In standard
> Solr search handlers, the default type of the main query only may be
> specified via the defType parameter. The default type of all other query
> parameters will remain "lucene
> <http://wiki.apache.org/solr/SolrQuerySyntax#lucene>"."
>
> I _think_ it's trying to say that I _can_, even in a standard search
> handler, force dismax with {!dismax}, I just can't change the type of
> _other_ query parameters -- rather than saying that I _can't_ use {!dismax}
> to force dismax type of 'q' in a "standard search handler".  Yes?



> Next, when we get to dismax---can I embed parameter value literals in there,
> and how?
>
> q={!dismax qf="field^2 field2^4"} some query
>
> With quotes like that? What if a parameter value literal itself has a quote
> mark in it, is there a way to escape it? (Say, can I embed an fq in there
> whose value itself needs quotes?)
>
> Will parameters supplied in LocalParams like that always "override"
> something supplied in a straight query parameter?
>
> &qf=somefield&q={!dismax qf=otherfield} cat dog    => I'll get "otherfield"
> and not "somefield" ?
>
> Can you use the LocalParams _without_ a !defType, to just supply parameters
> but take the default defType for the request handler?    &q={qf=somefield}
> cat dog
>
> And now we move into the nested querries -- for the simplest case nested
> query, is it in fact identical to using LocalParams without a nested query?
>
> &q={!dismax qf=something} cat dog
>
> ===?
>
> &q=_query_:"{!dismax qf=somefield} cat dog"
>
> Are those equivelent?  In nested querries like that, the issue of quote
> escaping becomes even more confusing, can I put a literal in there, and...
> how?
>
> &q=_query_:"{!dismax qf="somefield^1 otherfield^2"}"   ==> obviously won't
> work, the quotes are interfering with each other.
>
> Phew, that's a lot of questions, but probably just a start, heh. I'm trying
> to wrap my head around the complete possibilities of LocalParams and nested
> querries, to figure out if I can use them to do the kind of complicated
> thing I'm trying to do.
> I guess I could try to test all these things out and figure out the answer,
> and probably will proceed to do that if nobody else has any other advice,
> but I figured it didn't hurt to ask, and some of these things will be hard
> to guess on my own (like the quote escaping issue).  If I figure some stuff
> out, is there any good community place (wiki somewhere?) for me to document
> this stuff for others?
>
> Thanks for any advice,
>
> Jonathan
>
>
Reply | Threaded
Open this post in threaded view
|

Re: nested querries, and LocalParams syntax

Chris Hostetter-3
In reply to this post by Jonathan Rochkind

In addition to yonik's point about the LocalParams wiki page (and please
let us know if you aren't sure of the answers to any of your questions
after reading it) I wanted to clear up one thing...

: Let's start with that not-nested query example.   Can you in fact use it as
: above, to force dismax handling of the 'q' even if the qt or request handler

Quick side note: "qt" determines the ReequestHandler -- if it's "dismax"
then you get the DisMaxRequestHandler which in recent versions of solr is
just a thin subclass of the SearchHandler subclass where the
default value of "defType" (which is used to pick a QParser) is "dismax"
instead of "lucene" ... i tried to explain this in a recent blog...

http://www.lucidimagination.com/blog/2010/05/23/whats-a-dismax/

... the key thing to note is that "defType" is a param that is specific to
SearchHandler -- if you use "qt" to pick some other third party
RequestHandler, it's not neccessarily going to do *anything* and the
nested params syntax may not work at all.

: default is something else?  The documentation is confusing: "In standard Solr
: search handlers, the default type of the main query only may be specified via
: the defType parameter. The default type of all other query parameters will
: remain "lucene <http://wiki.apache.org/solr/SolrQuerySyntax#lucene>"."
:
: I _think_ it's trying to say that I _can_, even in a standard search handler,
: force dismax with {!dismax}, I just can't change the type of _other_ query
: parameters -- rather than saying that I _can't_ use {!dismax} to force dismax
: type of 'q' in a "standard search handler".  Yes?

You're right, it is confusing -- the point is tha defType changes the
"default QParser type" for the "q" param -- but it doesn't change it for
any other param.  I've improved the wording, but the key to keep in mind
is that that is completley orthoginal to using the local params syntax
that you asked about.

What that documentation is trying to illustrate is that in this request...

   defType=XXX&q=AAA&fq=BBB

...the "XXX" QParser will be used to parse the value "AAA" -- but the
stock "lucene" QParser will be used to parse the "fq" param

Regardless of the value of defType, if you put the local params
syntax ({!foo}) at the begining of a query param, you can force that param
to be parsed the way you wish...

   defType=XXX&q={!foo}AAA&fq={!bar}BBB

...in that example, neither the XXX or "lucene" QParsers are ever used.



-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: nested querries, and LocalParams syntax

Jonathan Rochkind
Thanks, the pointer to that documentation page (which somehow I had
missed), as well as Chris's response is very helpful.

The one thing I'm still not sure about, which I might be able to figure
it out through trial-and-error reverse engineering, is escaping issues
when you combine nested querries WITH local params. We potentially have
a lot of levels of quotes:

q= URIescape(    _local_="{!dismax qf=" value that itself contains a \"
quote mark"} "phrase query"    "   )

Whole bunch of quotes going on there. How do I give this to Solr so all
my quotes will end up parsed appropriately? Obviously that above example
isn't right.   We've got the quotes around the _local_ nested query,
then we've got quotes around a LocalParam value, then we've got quotes
that might be IN the actual literal value of the LocalParam, or quotes
that might be in the actual literal value of the nested query.  Maybe
using single quotes in some places but double quotes in others will
help, for certain places that can take singel or double quotes?

Thanks very much for any advice, I get confused thinking about this.

Jonathan****

Chris Hostetter wrote:

> In addition to yonik's point about the LocalParams wiki page (and please
> let us know if you aren't sure of the answers to any of your questions
> after reading it) I wanted to clear up one thing...
>
> : Let's start with that not-nested query example.   Can you in fact use it as
> : above, to force dismax handling of the 'q' even if the qt or request handler
>
> Quick side note: "qt" determines the ReequestHandler -- if it's "dismax"
> then you get the DisMaxRequestHandler which in recent versions of solr is
> just a thin subclass of the SearchHandler subclass where the
> default value of "defType" (which is used to pick a QParser) is "dismax"
> instead of "lucene" ... i tried to explain this in a recent blog...
>
> http://www.lucidimagination.com/blog/2010/05/23/whats-a-dismax/
>
> ... the key thing to note is that "defType" is a param that is specific to
> SearchHandler -- if you use "qt" to pick some other third party
> RequestHandler, it's not neccessarily going to do *anything* and the
> nested params syntax may not work at all.
>
> : default is something else?  The documentation is confusing: "In standard Solr
> : search handlers, the default type of the main query only may be specified via
> : the defType parameter. The default type of all other query parameters will
> : remain "lucene <http://wiki.apache.org/solr/SolrQuerySyntax#lucene>"."
> :
> : I _think_ it's trying to say that I _can_, even in a standard search handler,
> : force dismax with {!dismax}, I just can't change the type of _other_ query
> : parameters -- rather than saying that I _can't_ use {!dismax} to force dismax
> : type of 'q' in a "standard search handler".  Yes?
>
> You're right, it is confusing -- the point is tha defType changes the
> "default QParser type" for the "q" param -- but it doesn't change it for
> any other param.  I've improved the wording, but the key to keep in mind
> is that that is completley orthoginal to using the local params syntax
> that you asked about.
>
> What that documentation is trying to illustrate is that in this request...
>
>    defType=XXX&q=AAA&fq=BBB
>
> ...the "XXX" QParser will be used to parse the value "AAA" -- but the
> stock "lucene" QParser will be used to parse the "fq" param
>
> Regardless of the value of defType, if you put the local params
> syntax ({!foo}) at the begining of a query param, you can force that param
> to be parsed the way you wish...
>
>    defType=XXX&q={!foo}AAA&fq={!bar}BBB
>
> ...in that example, neither the XXX or "lucene" QParsers are ever used.
>
>
>
> -Hoss
>
>
>  
Reply | Threaded
Open this post in threaded view
|

Re: nested querries, and LocalParams syntax

Yonik Seeley-2-2
It's not clear if you're just trying to figure it all out, or get
something specific to work.
If you can give a specific example, we might be able to suggest easier
ways to achieve it rather than going escape crazy :-)

-Yonik
http://www.lucidimagination.com



On Tue, Jun 1, 2010 at 5:06 PM, Jonathan Rochkind <[hidden email]> wrote:

> Thanks, the pointer to that documentation page (which somehow I had missed),
> as well as Chris's response is very helpful.
>
> The one thing I'm still not sure about, which I might be able to figure it
> out through trial-and-error reverse engineering, is escaping issues when you
> combine nested querries WITH local params. We potentially have a lot of
> levels of quotes:
>
> q= URIescape(    _local_="{!dismax qf=" value that itself contains a \"
> quote mark"} "phrase query"    "   )
>
> Whole bunch of quotes going on there. How do I give this to Solr so all my
> quotes will end up parsed appropriately? Obviously that above example isn't
> right.   We've got the quotes around the _local_ nested query, then we've
> got quotes around a LocalParam value, then we've got quotes that might be IN
> the actual literal value of the LocalParam, or quotes that might be in the
> actual literal value of the nested query.  Maybe using single quotes in some
> places but double quotes in others will help, for certain places that can
> take singel or double quotes?
> Thanks very much for any advice, I get confused thinking about this.
>
> Jonathan****
>
> Chris Hostetter wrote:
>>
>> In addition to yonik's point about the LocalParams wiki page (and please
>> let us know if you aren't sure of the answers to any of your questions after
>> reading it) I wanted to clear up one thing...
>>
>> : Let's start with that not-nested query example.   Can you in fact use it
>> as
>> : above, to force dismax handling of the 'q' even if the qt or request
>> handler
>>
>> Quick side note: "qt" determines the ReequestHandler -- if it's "dismax"
>> then you get the DisMaxRequestHandler which in recent versions of solr is
>> just a thin subclass of the SearchHandler subclass where the default value
>> of "defType" (which is used to pick a QParser) is "dismax" instead of
>> "lucene" ... i tried to explain this in a recent blog...
>>
>> http://www.lucidimagination.com/blog/2010/05/23/whats-a-dismax/
>>
>> ... the key thing to note is that "defType" is a param that is specific to
>> SearchHandler -- if you use "qt" to pick some other third party
>> RequestHandler, it's not neccessarily going to do *anything* and the nested
>> params syntax may not work at all.
>>
>> : default is something else?  The documentation is confusing: "In standard
>> Solr
>> : search handlers, the default type of the main query only may be
>> specified via
>> : the defType parameter. The default type of all other query parameters
>> will
>> : remain "lucene <http://wiki.apache.org/solr/SolrQuerySyntax#lucene>"."
>> : : I _think_ it's trying to say that I _can_, even in a standard search
>> handler,
>> : force dismax with {!dismax}, I just can't change the type of _other_
>> query
>> : parameters -- rather than saying that I _can't_ use {!dismax} to force
>> dismax
>> : type of 'q' in a "standard search handler".  Yes?
>>
>> You're right, it is confusing -- the point is tha defType changes the
>> "default QParser type" for the "q" param -- but it doesn't change it for any
>> other param.  I've improved the wording, but the key to keep in mind is that
>> that is completley orthoginal to using the local params syntax that you
>> asked about.
>>
>> What that documentation is trying to illustrate is that in this request...
>>
>>   defType=XXX&q=AAA&fq=BBB
>>
>> ...the "XXX" QParser will be used to parse the value "AAA" -- but the
>> stock "lucene" QParser will be used to parse the "fq" param
>>
>> Regardless of the value of defType, if you put the local params syntax
>> ({!foo}) at the begining of a query param, you can force that param to be
>> parsed the way you wish...
>>
>>   defType=XXX&q={!foo}AAA&fq={!bar}BBB
>>
>> ...in that example, neither the XXX or "lucene" QParsers are ever used.
>>
>>
>>
>> -Hoss
>>
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: nested querries, and LocalParams syntax

Jonathan Rochkind
I am just trying to figure it out mostly, the particular thing I am
trying to do is a very general purpose mapper to complex dismax nested
querries.  I could try to explain it, and we could go back and forth for
a while, and maybe I could convince you it makes sense to do what I'm
trying to do.  But mostly I'm just exploring at this point, so I can get
a sense of what is possible.

So it would be super helpful if someone can help me figure out escaping
stuff and skip the other part, heh.

But basically, it's a mapper from a "CQL" query (a structured language
for search-engine-style querries) to Solr, where some of the "fields"
searched aren't really Solr fields/indexes, but aggregated definitions
of dismax query params including multiple solr fields, where exactly
what solr fields and other dismax querries will not be hard-coded, but
will be configurable.  Thus the use of nested querries. So since it ends
up so general purpose and abstract, and many of the individual
parameters are configurable, thus my interest in figuring out proper
escaping.

Jonathan

Yonik Seeley wrote:

> It's not clear if you're just trying to figure it all out, or get
> something specific to work.
> If you can give a specific example, we might be able to suggest easier
> ways to achieve it rather than going escape crazy :-)
>
> -Yonik
> http://www.lucidimagination.com
>
>
>
> On Tue, Jun 1, 2010 at 5:06 PM, Jonathan Rochkind <[hidden email]> wrote:
>  
>> Thanks, the pointer to that documentation page (which somehow I had missed),
>> as well as Chris's response is very helpful.
>>
>> The one thing I'm still not sure about, which I might be able to figure it
>> out through trial-and-error reverse engineering, is escaping issues when you
>> combine nested querries WITH local params. We potentially have a lot of
>> levels of quotes:
>>
>> q= URIescape(    _local_="{!dismax qf=" value that itself contains a \"
>> quote mark"} "phrase query"    "   )
>>
>> Whole bunch of quotes going on there. How do I give this to Solr so all my
>> quotes will end up parsed appropriately? Obviously that above example isn't
>> right.   We've got the quotes around the _local_ nested query, then we've
>> got quotes around a LocalParam value, then we've got quotes that might be IN
>> the actual literal value of the LocalParam, or quotes that might be in the
>> actual literal value of the nested query.  Maybe using single quotes in some
>> places but double quotes in others will help, for certain places that can
>> take singel or double quotes?
>> Thanks very much for any advice, I get confused thinking about this.
>>
>> Jonathan****
>>
>> Chris Hostetter wrote:
>>    
>>> In addition to yonik's point about the LocalParams wiki page (and please
>>> let us know if you aren't sure of the answers to any of your questions after
>>> reading it) I wanted to clear up one thing...
>>>
>>> : Let's start with that not-nested query example.   Can you in fact use it
>>> as
>>> : above, to force dismax handling of the 'q' even if the qt or request
>>> handler
>>>
>>> Quick side note: "qt" determines the ReequestHandler -- if it's "dismax"
>>> then you get the DisMaxRequestHandler which in recent versions of solr is
>>> just a thin subclass of the SearchHandler subclass where the default value
>>> of "defType" (which is used to pick a QParser) is "dismax" instead of
>>> "lucene" ... i tried to explain this in a recent blog...
>>>
>>> http://www.lucidimagination.com/blog/2010/05/23/whats-a-dismax/
>>>
>>> ... the key thing to note is that "defType" is a param that is specific to
>>> SearchHandler -- if you use "qt" to pick some other third party
>>> RequestHandler, it's not neccessarily going to do *anything* and the nested
>>> params syntax may not work at all.
>>>
>>> : default is something else?  The documentation is confusing: "In standard
>>> Solr
>>> : search handlers, the default type of the main query only may be
>>> specified via
>>> : the defType parameter. The default type of all other query parameters
>>> will
>>> : remain "lucene <http://wiki.apache.org/solr/SolrQuerySyntax#lucene>"."
>>> : : I _think_ it's trying to say that I _can_, even in a standard search
>>> handler,
>>> : force dismax with {!dismax}, I just can't change the type of _other_
>>> query
>>> : parameters -- rather than saying that I _can't_ use {!dismax} to force
>>> dismax
>>> : type of 'q' in a "standard search handler".  Yes?
>>>
>>> You're right, it is confusing -- the point is tha defType changes the
>>> "default QParser type" for the "q" param -- but it doesn't change it for any
>>> other param.  I've improved the wording, but the key to keep in mind is that
>>> that is completley orthoginal to using the local params syntax that you
>>> asked about.
>>>
>>> What that documentation is trying to illustrate is that in this request...
>>>
>>>   defType=XXX&q=AAA&fq=BBB
>>>
>>> ...the "XXX" QParser will be used to parse the value "AAA" -- but the
>>> stock "lucene" QParser will be used to parse the "fq" param
>>>
>>> Regardless of the value of defType, if you put the local params syntax
>>> ({!foo}) at the begining of a query param, you can force that param to be
>>> parsed the way you wish...
>>>
>>>   defType=XXX&q={!foo}AAA&fq={!bar}BBB
>>>
>>> ...in that example, neither the XXX or "lucene" QParsers are ever used.
>>>
>>>
>>>
>>> -Hoss
>>>
>>>
>>>
>>>      
>
>  
Reply | Threaded
Open this post in threaded view
|

Re: nested querries, and LocalParams syntax

Yonik Seeley-2-2
Hmmm, well, the lucene query parser does basic backslash escaping, and
so does local params within quoted strings.  You can also use
parameter derefererencing to avoid the need to escape values too.
Like you pointed out, using single quotes in some places can also
help.

But instead of me trying to give you tons of examples that you
probably already understand, start from the assumption that things
will work, and if you come across something that doesn't make sense
(or doesn't work), I can help with that.   Or if you give a single
real example as a general pattern, perhaps we could help figure out
the simplest way to avoid most of the escaping.

-Yonik
http://www.lucidimagination.com



On Tue, Jun 1, 2010 at 6:21 PM, Jonathan Rochkind <[hidden email]> wrote:

> I am just trying to figure it out mostly, the particular thing I am trying
> to do is a very general purpose mapper to complex dismax nested querries.  I
> could try to explain it, and we could go back and forth for a while, and
> maybe I could convince you it makes sense to do what I'm trying to do.  But
> mostly I'm just exploring at this point, so I can get a sense of what is
> possible.
>
> So it would be super helpful if someone can help me figure out escaping
> stuff and skip the other part, heh.
>
> But basically, it's a mapper from a "CQL" query (a structured language for
> search-engine-style querries) to Solr, where some of the "fields" searched
> aren't really Solr fields/indexes, but aggregated definitions of dismax
> query params including multiple solr fields, where exactly what solr fields
> and other dismax querries will not be hard-coded, but will be configurable.
>  Thus the use of nested querries. So since it ends up so general purpose and
> abstract, and many of the individual parameters are configurable, thus my
> interest in figuring out proper escaping.
>
> Jonathan
>
> Yonik Seeley wrote:
>>
>> It's not clear if you're just trying to figure it all out, or get
>> something specific to work.
>> If you can give a specific example, we might be able to suggest easier
>> ways to achieve it rather than going escape crazy :-)
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>>
>> On Tue, Jun 1, 2010 at 5:06 PM, Jonathan Rochkind <[hidden email]>
>> wrote:
>>
>>>
>>> Thanks, the pointer to that documentation page (which somehow I had
>>> missed),
>>> as well as Chris's response is very helpful.
>>>
>>> The one thing I'm still not sure about, which I might be able to figure
>>> it
>>> out through trial-and-error reverse engineering, is escaping issues when
>>> you
>>> combine nested querries WITH local params. We potentially have a lot of
>>> levels of quotes:
>>>
>>> q= URIescape(    _local_="{!dismax qf=" value that itself contains a \"
>>> quote mark"} "phrase query"    "   )
>>>
>>> Whole bunch of quotes going on there. How do I give this to Solr so all
>>> my
>>> quotes will end up parsed appropriately? Obviously that above example
>>> isn't
>>> right.   We've got the quotes around the _local_ nested query, then we've
>>> got quotes around a LocalParam value, then we've got quotes that might be
>>> IN
>>> the actual literal value of the LocalParam, or quotes that might be in
>>> the
>>> actual literal value of the nested query.  Maybe using single quotes in
>>> some
>>> places but double quotes in others will help, for certain places that can
>>> take singel or double quotes?
>>> Thanks very much for any advice, I get confused thinking about this.
>>>
>>> Jonathan****
>>>
>>> Chris Hostetter wrote:
>>>
>>>>
>>>> In addition to yonik's point about the LocalParams wiki page (and please
>>>> let us know if you aren't sure of the answers to any of your questions
>>>> after
>>>> reading it) I wanted to clear up one thing...
>>>>
>>>> : Let's start with that not-nested query example.   Can you in fact use
>>>> it
>>>> as
>>>> : above, to force dismax handling of the 'q' even if the qt or request
>>>> handler
>>>>
>>>> Quick side note: "qt" determines the ReequestHandler -- if it's "dismax"
>>>> then you get the DisMaxRequestHandler which in recent versions of solr
>>>> is
>>>> just a thin subclass of the SearchHandler subclass where the default
>>>> value
>>>> of "defType" (which is used to pick a QParser) is "dismax" instead of
>>>> "lucene" ... i tried to explain this in a recent blog...
>>>>
>>>> http://www.lucidimagination.com/blog/2010/05/23/whats-a-dismax/
>>>>
>>>> ... the key thing to note is that "defType" is a param that is specific
>>>> to
>>>> SearchHandler -- if you use "qt" to pick some other third party
>>>> RequestHandler, it's not neccessarily going to do *anything* and the
>>>> nested
>>>> params syntax may not work at all.
>>>>
>>>> : default is something else?  The documentation is confusing: "In
>>>> standard
>>>> Solr
>>>> : search handlers, the default type of the main query only may be
>>>> specified via
>>>> : the defType parameter. The default type of all other query parameters
>>>> will
>>>> : remain "lucene <http://wiki.apache.org/solr/SolrQuerySyntax#lucene>"."
>>>> : : I _think_ it's trying to say that I _can_, even in a standard search
>>>> handler,
>>>> : force dismax with {!dismax}, I just can't change the type of _other_
>>>> query
>>>> : parameters -- rather than saying that I _can't_ use {!dismax} to force
>>>> dismax
>>>> : type of 'q' in a "standard search handler".  Yes?
>>>>
>>>> You're right, it is confusing -- the point is tha defType changes the
>>>> "default QParser type" for the "q" param -- but it doesn't change it for
>>>> any
>>>> other param.  I've improved the wording, but the key to keep in mind is
>>>> that
>>>> that is completley orthoginal to using the local params syntax that you
>>>> asked about.
>>>>
>>>> What that documentation is trying to illustrate is that in this
>>>> request...
>>>>
>>>>  defType=XXX&q=AAA&fq=BBB
>>>>
>>>> ...the "XXX" QParser will be used to parse the value "AAA" -- but the
>>>> stock "lucene" QParser will be used to parse the "fq" param
>>>>
>>>> Regardless of the value of defType, if you put the local params syntax
>>>> ({!foo}) at the begining of a query param, you can force that param to
>>>> be
>>>> parsed the way you wish...
>>>>
>>>>  defType=XXX&q={!foo}AAA&fq={!bar}BBB
>>>>
>>>> ...in that example, neither the XXX or "lucene" QParsers are ever used.
>>>>
>>>>
>>>>
>>>> -Hoss
>>>>
>>>>
>>>>
>>>>
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: nested querries, and LocalParams syntax

Jonathan Rochkind
Thanks Yonik.

I guess the confusing thing is if the lucene query parser (for nested
querries) does backslash escaping, and the LocalParams also does
backslash escaping.... when you have a nested query with local params,
with quotes at both places... the inner scope needs... double escaping?  
That gets really confusing fast.

[ Yeah, I recognize that using parameter dereferencing can avoid this;
I'm trying to see if I can make my code flexible enough to work either
way].

Maybe using single vs double quotes is the answer. Let's try one out and
see:

[Query un-uri escaped for clarity:]

_query_:"{!dismax q.alt=' \"a phrase search \" '} \"another phrase
search\" "

[ Heh, getting that into a ruby string to uri escape it is a pain, but
we end up with: ]

&q="_query_%3A%7B%21dismax+q.alt%3D%27%5C%22a+phrase+search%5C%22%27%7D+%5C%22another+phrase+search%5C%22

Which, um, I _think_ is working, although the debugQuery=true isn't
telling me much, I don't entirely understand it. Have to play around
with it more.

But it looks like maybe a fine strategy is use double quote for the
nested query itself, use single quote for the LocalParam values, and
then simply singly escape any single or double quotes inside the
LocalParam values.

Jonathan


Yonik Seeley wrote:

> Hmmm, well, the lucene query parser does basic backslash escaping, and
> so does local params within quoted strings.  You can also use
> parameter derefererencing to avoid the need to escape values too.
> Like you pointed out, using single quotes in some places can also
> help.
>
> But instead of me trying to give you tons of examples that you
> probably already understand, start from the assumption that things
> will work, and if you come across something that doesn't make sense
> (or doesn't work), I can help with that.   Or if you give a single
> real example as a general pattern, perhaps we could help figure out
> the simplest way to avoid most of the escaping.
>
> -Yonik
> http://www.lucidimagination.com
>
>
>
> On Tue, Jun 1, 2010 at 6:21 PM, Jonathan Rochkind <[hidden email]> wrote:
>  
>> I am just trying to figure it out mostly, the particular thing I am trying
>> to do is a very general purpose mapper to complex dismax nested querries.  I
>> could try to explain it, and we could go back and forth for a while, and
>> maybe I could convince you it makes sense to do what I'm trying to do.  But
>> mostly I'm just exploring at this point, so I can get a sense of what is
>> possible.
>>
>> So it would be super helpful if someone can help me figure out escaping
>> stuff and skip the other part, heh.
>>
>> But basically, it's a mapper from a "CQL" query (a structured language for
>> search-engine-style querries) to Solr, where some of the "fields" searched
>> aren't really Solr fields/indexes, but aggregated definitions of dismax
>> query params including multiple solr fields, where exactly what solr fields
>> and other dismax querries will not be hard-coded, but will be configurable.
>>  Thus the use of nested querries. So since it ends up so general purpose and
>> abstract, and many of the individual parameters are configurable, thus my
>> interest in figuring out proper escaping.
>>
>> Jonathan
>>
>> Yonik Seeley wrote:
>>    
>>> It's not clear if you're just trying to figure it all out, or get
>>> something specific to work.
>>> If you can give a specific example, we might be able to suggest easier
>>> ways to achieve it rather than going escape crazy :-)
>>>
>>> -Yonik
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>> On Tue, Jun 1, 2010 at 5:06 PM, Jonathan Rochkind <[hidden email]>
>>> wrote:
>>>
>>>      
>>>> Thanks, the pointer to that documentation page (which somehow I had
>>>> missed),
>>>> as well as Chris's response is very helpful.
>>>>
>>>> The one thing I'm still not sure about, which I might be able to figure
>>>> it
>>>> out through trial-and-error reverse engineering, is escaping issues when
>>>> you
>>>> combine nested querries WITH local params. We potentially have a lot of
>>>> levels of quotes:
>>>>
>>>> q= URIescape(    _local_="{!dismax qf=" value that itself contains a \"
>>>> quote mark"} "phrase query"    "   )
>>>>
>>>> Whole bunch of quotes going on there. How do I give this to Solr so all
>>>> my
>>>> quotes will end up parsed appropriately? Obviously that above example
>>>> isn't
>>>> right.   We've got the quotes around the _local_ nested query, then we've
>>>> got quotes around a LocalParam value, then we've got quotes that might be
>>>> IN
>>>> the actual literal value of the LocalParam, or quotes that might be in
>>>> the
>>>> actual literal value of the nested query.  Maybe using single quotes in
>>>> some
>>>> places but double quotes in others will help, for certain places that can
>>>> take singel or double quotes?
>>>> Thanks very much for any advice, I get confused thinking about this.
>>>>
>>>> Jonathan****
>>>>
>>>> Chris Hostetter wrote:
>>>>
>>>>        
>>>>> In addition to yonik's point about the LocalParams wiki page (and please
>>>>> let us know if you aren't sure of the answers to any of your questions
>>>>> after
>>>>> reading it) I wanted to clear up one thing...
>>>>>
>>>>> : Let's start with that not-nested query example.   Can you in fact use
>>>>> it
>>>>> as
>>>>> : above, to force dismax handling of the 'q' even if the qt or request
>>>>> handler
>>>>>
>>>>> Quick side note: "qt" determines the ReequestHandler -- if it's "dismax"
>>>>> then you get the DisMaxRequestHandler which in recent versions of solr
>>>>> is
>>>>> just a thin subclass of the SearchHandler subclass where the default
>>>>> value
>>>>> of "defType" (which is used to pick a QParser) is "dismax" instead of
>>>>> "lucene" ... i tried to explain this in a recent blog...
>>>>>
>>>>> http://www.lucidimagination.com/blog/2010/05/23/whats-a-dismax/
>>>>>
>>>>> ... the key thing to note is that "defType" is a param that is specific
>>>>> to
>>>>> SearchHandler -- if you use "qt" to pick some other third party
>>>>> RequestHandler, it's not neccessarily going to do *anything* and the
>>>>> nested
>>>>> params syntax may not work at all.
>>>>>
>>>>> : default is something else?  The documentation is confusing: "In
>>>>> standard
>>>>> Solr
>>>>> : search handlers, the default type of the main query only may be
>>>>> specified via
>>>>> : the defType parameter. The default type of all other query parameters
>>>>> will
>>>>> : remain "lucene <http://wiki.apache.org/solr/SolrQuerySyntax#lucene>"."
>>>>> : : I _think_ it's trying to say that I _can_, even in a standard search
>>>>> handler,
>>>>> : force dismax with {!dismax}, I just can't change the type of _other_
>>>>> query
>>>>> : parameters -- rather than saying that I _can't_ use {!dismax} to force
>>>>> dismax
>>>>> : type of 'q' in a "standard search handler".  Yes?
>>>>>
>>>>> You're right, it is confusing -- the point is tha defType changes the
>>>>> "default QParser type" for the "q" param -- but it doesn't change it for
>>>>> any
>>>>> other param.  I've improved the wording, but the key to keep in mind is
>>>>> that
>>>>> that is completley orthoginal to using the local params syntax that you
>>>>> asked about.
>>>>>
>>>>> What that documentation is trying to illustrate is that in this
>>>>> request...
>>>>>
>>>>>  defType=XXX&q=AAA&fq=BBB
>>>>>
>>>>> ...the "XXX" QParser will be used to parse the value "AAA" -- but the
>>>>> stock "lucene" QParser will be used to parse the "fq" param
>>>>>
>>>>> Regardless of the value of defType, if you put the local params syntax
>>>>> ({!foo}) at the begining of a query param, you can force that param to
>>>>> be
>>>>> parsed the way you wish...
>>>>>
>>>>>  defType=XXX&q={!foo}AAA&fq={!bar}BBB
>>>>>
>>>>> ...in that example, neither the XXX or "lucene" QParsers are ever used.
>>>>>
>>>>>
>>>>>
>>>>> -Hoss
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>          
>>>      
>
>