how to best convert some term in q to a fq

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

how to best convert some term in q to a fq

jmlucjav
Hi,

I have this scenario that I think is no unusual: solr will get a user
entered query string like 'apple pear france'.

I need to do this: if any of the terms is a country, then change the query
params to move that term to a fq, i.e:
q=apple pear france
to
q=apple pear&fq=country:france

What do you guys would be the best way to implement this?
- custom searchcomponent or queryparser
- servlet in same jetty as solr
- client code

To simplify, consider countries are just a single term.

Any pointer to an example to base this on would be great. thanks
Reply | Threaded
Open this post in threaded view
|

Re: how to best convert some term in q to a fq

Joel Bernstein
I  would suggest handling this in the client. You could write custom Solr
code also but it would be more complicated because you'd be working with
Solr's API's.

Joel Bernstein
Search Engineer at Heliosearch


On Mon, Dec 23, 2013 at 2:36 PM, jmlucjav <[hidden email]> wrote:

> Hi,
>
> I have this scenario that I think is no unusual: solr will get a user
> entered query string like 'apple pear france'.
>
> I need to do this: if any of the terms is a country, then change the query
> params to move that term to a fq, i.e:
> q=apple pear france
> to
> q=apple pear&fq=country:france
>
> What do you guys would be the best way to implement this?
> - custom searchcomponent or queryparser
> - servlet in same jetty as solr
> - client code
>
> To simplify, consider countries are just a single term.
>
> Any pointer to an example to base this on would be great. thanks
>
Reply | Threaded
Open this post in threaded view
|

Re: how to best convert some term in q to a fq

jlman
what if you add your country field to qf with a strong boost? the search
experience would be slightly different than if you filter on country, but
maybe still good enough for your users and certainly simpler to implement
and maintain. You'd likely only want exact matches. Assuming you are using
edismax and a stopword file for your main query fields, you'll run into an
issue if you just index your country field as a string and there's a
stopword anywhere in your query...see SOLR-3085. To avoid this, yet still
boost on country only when there's an exact match, you could index the
country field as text using KeywordTokenizerFactory and the same stopword
file as your other fields.

Regardless of the approach you take, unless there's only a small list of
countries you care about, multi-word countries might be too big an issue to
ignore, especially when the name contains common words (e.g. United States,
South Korea, New Zealand). This may be a good candidate for named entity
recognition on the query, possibly leveraging openNLP. I once saw a
presentation on how linkedin uses nlp on the query to detect the types of
entities the user is looking for. Seems similar to what you're trying to
accomplish. Of course, if countries are the only thing you're interested in
then you may be able to get away with client code for simple substring
matching using a static list of countries.

 On Dec 23, 2013 3:08 PM, "Joel Bernstein" <[hidden email]> wrote:

> I  would suggest handling this in the client. You could write custom Solr
> code also but it would be more complicated because you'd be working with
> Solr's API's.
>
> Joel Bernstein
> Search Engineer at Heliosearch
>
>
> On Mon, Dec 23, 2013 at 2:36 PM, jmlucjav <[hidden email]> wrote:
>
> > Hi,
> >
> > I have this scenario that I think is no unusual: solr will get a user
> > entered query string like 'apple pear france'.
> >
> > I need to do this: if any of the terms is a country, then change the
> query
> > params to move that term to a fq, i.e:
> > q=apple pear france
> > to
> > q=apple pear&fq=country:france
> >
> > What do you guys would be the best way to implement this?
> > - custom searchcomponent or queryparser
> > - servlet in same jetty as solr
> > - client code
> >
> > To simplify, consider countries are just a single term.
> >
> > Any pointer to an example to base this on would be great. thanks
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: how to best convert some term in q to a fq

Jason Hellman
In reply to this post by Joel Bernstein
I second this notion.

My reasoning focuses mostly on maintainability, where I posit that your client code will be far easier to extend/modify/troubleshoot than any effort spent attempting to do this within Solr.

Jason


On Dec 23, 2013, at 12:07 PM, Joel Bernstein <[hidden email]> wrote:

> I  would suggest handling this in the client. You could write custom Solr
> code also but it would be more complicated because you'd be working with
> Solr's API's.
>
> Joel Bernstein
> Search Engineer at Heliosearch
>
>
> On Mon, Dec 23, 2013 at 2:36 PM, jmlucjav <[hidden email]> wrote:
>
>> Hi,
>>
>> I have this scenario that I think is no unusual: solr will get a user
>> entered query string like 'apple pear france'.
>>
>> I need to do this: if any of the terms is a country, then change the query
>> params to move that term to a fq, i.e:
>> q=apple pear france
>> to
>> q=apple pear&fq=country:france
>>
>> What do you guys would be the best way to implement this?
>> - custom searchcomponent or queryparser
>> - servlet in same jetty as solr
>> - client code
>>
>> To simplify, consider countries are just a single term.
>>
>> Any pointer to an example to base this on would be great. thanks
>>

Reply | Threaded
Open this post in threaded view
|

Re: how to best convert some term in q to a fq

Kranti Parisa
In reply to this post by Joel Bernstein
As Joel suggested it would be easy to find that outside of solr. But if you really want to do the solr way, then you would need to write custom request handler or query component. The later needs more understanding of solr/lucene APIs. The former would be relatively simple to set the fq into solr params exists in solr query request.



> On Dec 23, 2013, at 3:07 PM, Joel Bernstein <[hidden email]> wrote:
>
> I  would suggest handling this in the client. You could write custom Solr
> code also but it would be more complicated because you'd be working with
> Solr's API's.
>
> Joel Bernstein
> Search Engineer at Heliosearch
>
>
>> On Mon, Dec 23, 2013 at 2:36 PM, jmlucjav <[hidden email]> wrote:
>>
>> Hi,
>>
>> I have this scenario that I think is no unusual: solr will get a user
>> entered query string like 'apple pear france'.
>>
>> I need to do this: if any of the terms is a country, then change the query
>> params to move that term to a fq, i.e:
>> q=apple pear france
>> to
>> q=apple pear&fq=country:france
>>
>> What do you guys would be the best way to implement this?
>> - custom searchcomponent or queryparser
>> - servlet in same jetty as solr
>> - client code
>>
>> To simplify, consider countries are just a single term.
>>
>> Any pointer to an example to base this on would be great. thanks
>>
Reply | Threaded
Open this post in threaded view
|

Re: how to best convert some term in q to a fq

Kranti Parisa
In reply to this post by jlman
I thought the intention to convert country as fq is to hit the filter cache for performance than boosting results.

> On Dec 27, 2013, at 10:31 PM, Josh Lincoln <[hidden email]> wrote:
>
> what if you add your country field to qf with a strong boost? the search
> experience would be slightly different than if you filter on country, but
> maybe still good enough for your users and certainly simpler to implement
> and maintain. You'd likely only want exact matches. Assuming you are using
> edismax and a stopword file for your main query fields, you'll run into an
> issue if you just index your country field as a string and there's a
> stopword anywhere in your query...see SOLR-3085. To avoid this, yet still
> boost on country only when there's an exact match, you could index the
> country field as text using KeywordTokenizerFactory and the same stopword
> file as your other fields.
>
> Regardless of the approach you take, unless there's only a small list of
> countries you care about, multi-word countries might be too big an issue to
> ignore, especially when the name contains common words (e.g. United States,
> South Korea, New Zealand). This may be a good candidate for named entity
> recognition on the query, possibly leveraging openNLP. I once saw a
> presentation on how linkedin uses nlp on the query to detect the types of
> entities the user is looking for. Seems similar to what you're trying to
> accomplish. Of course, if countries are the only thing you're interested in
> then you may be able to get away with client code for simple substring
> matching using a static list of countries.
>
>> On Dec 23, 2013 3:08 PM, "Joel Bernstein" <[hidden email]> wrote:
>>
>> I  would suggest handling this in the client. You could write custom Solr
>> code also but it would be more complicated because you'd be working with
>> Solr's API's.
>>
>> Joel Bernstein
>> Search Engineer at Heliosearch
>>
>>
>>> On Mon, Dec 23, 2013 at 2:36 PM, jmlucjav <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> I have this scenario that I think is no unusual: solr will get a user
>>> entered query string like 'apple pear france'.
>>>
>>> I need to do this: if any of the terms is a country, then change the
>> query
>>> params to move that term to a fq, i.e:
>>> q=apple pear france
>>> to
>>> q=apple pear&fq=country:france
>>>
>>> What do you guys would be the best way to implement this?
>>> - custom searchcomponent or queryparser
>>> - servlet in same jetty as solr
>>> - client code
>>>
>>> To simplify, consider countries are just a single term.
>>>
>>> Any pointer to an example to base this on would be great. thanks
>>