Problem with white space or special characters in function queries

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with white space or special characters in function queries

shamik
This post was updated on .
Hi,

   I'm having trouble handling white space or special characters in
function queries. Here's a sample function:

if(termfreq(ADSKFeature,'CUI (Command)'),log(CaseCount),sqrt(CaseCount))

I tried escaping the space with "\", but that didn't work either. Here's
the exception being thrown:

org.apache.solr.search.SyntaxError: Expected identifier at pos 0
str='(Command)'),log(CaseCount),sqrt(CaseCount))'

This happen with other special characters like &, ?, etc.

Does function queries have any limitations? Or am I doing something wrong? I'm using Solr 7.5, ADSKFeature field definition :

<field name="ADSKFeature" type="string" indexed="true" stored="true" multiValued="true" required="false" omitNorms="true"  docValues="true" />

Any pointers will be highly appreciated.

Thanks,
Shamik
Reply | Threaded
Open this post in threaded view
|

Re: Problem with white space or special characters in function queries

Zheng Lin Edwin Yeo
Hi Shamik,

What fieldType are you using for the field that you are doing the function
queries on?

Regards,
Edwin

On Tue, 26 Mar 2019 at 06:40, Shamik Bandopadhyay <[hidden email]> wrote:

> Hi,
>
>    I'm having trouble handling white space or special characters in
> function queries. Here's a sample function:
>
> if(termfreq(ADSKFeature,'CUI (Command)'),log(CaseCount),sqrt(CaseCount))
>
> I tried escaping the space with "\", but that didn't work either. Here's
> the exception being thrown:
>
> org.apache.solr.search.SyntaxError: Expected identifier at pos 0
> str='(Command)'),log(CaseCount),sqrt(CaseCount))'
>
> This happen with other special characters like &, ?, etc.
>
> Does function queries have any limitations? Or am I doing something wrong?
>
> Any pointers will be highly appreciated.
>
> Thanks,
> Shamik
>
Reply | Threaded
Open this post in threaded view
|

Re: Problem with white space or special characters in function queries

shamik
Edwin,

   The field is a string type, here's the field definition.

<field name="ADSKFeature" type="string" indexed="true" stored="true"
multiValued="true" required="false" omitNorms="true"  docValues="true" />

-Shamik



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Problem with white space or special characters in function queries

Zheng Lin Edwin Yeo
Hi Shamik,

String fieldType should keep the space. Can you give the full query that
you used when you get the error?

Also, which Solr version are you using?

Regards,
Edwin

On Wed, 27 Mar 2019 at 03:16, shamik <[hidden email]> wrote:

> Edwin,
>
>    The field is a string type, here's the field definition.
>
> <field name="ADSKFeature" type="string" indexed="true" stored="true"
> multiValued="true" required="false" omitNorms="true"  docValues="true" />
>
> -Shamik
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Problem with white space or special characters in function queries

shamik
I'm using Solr 7.5, here's the query:

q=line&fq=language:"english"&fq=Source2:("topicarticles"+OR+"sfdcarticles")&fl=url,title&bq=ADSKFeature:"CUI+(Command)"^7&bf=recip(ms(NOW/DAY,PublishDate),3.16e-11,1,1)^2+if(termfreq(ADSKFeature,'CUI
(Command)'),log(CaseCount),sqrt(CaseCount))&rows=10



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Problem with white space or special characters in function queries

Ahemad Ali
Hi,Did you find any solution for this, I tried adding analyzers with keyword tokenizer and reversewildcardfilterfactory but was not working. 
It's working when I don't include spaces when I am doing wildcard search but not giving the result when I have whitespaces and do wildcard search using * and ?.
Any help is greatly appreciated.
Thanks,Ahemad Ali Shaik


Sent from Yahoo Mail on Android
 
  On Thu, Mar 28, 2019 at 0:08, shamik<[hidden email]> wrote:   I'm using Solr 7.5, here's the query:

q=line&fq=language:"english"&fq=Source2:("topicarticles"+OR+"sfdcarticles")&fl=url,title&bq=ADSKFeature:"CUI+(Command)"^7&bf=recip(ms(NOW/DAY,PublishDate),3.16e-11,1,1)^2+if(termfreq(ADSKFeature,'CUI
(Command)'),log(CaseCount),sqrt(CaseCount))&rows=10



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
 
Reply | Threaded
Open this post in threaded view
|

Re: Problem with white space or special characters in function queries

shamik
Ahemad, I don't think its related to the field definition, rather looks like
an inherent bug. For the time being, I created a copyfield which uses a
custom regex to remove whitespace and special characters and use it in the
function. I'll debug the source code and confirm if it's bug, will raise a
JIRA if needed.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Problem with white space or special characters in function queries

Jan Høydahl / Cominvent
Functions can never contain spaces. Try to substitute the term with a variable, i.e. a request parameter, e.g.

bf=if(termfreq(ADSKFeature,$myTerm),log(CaseCount),sqrt(CaseCount))&myTerm=CUI+(Command)

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 28. mar. 2019 kl. 18:51 skrev shamik <[hidden email]>:
>
> Ahemad, I don't think its related to the field definition, rather looks like
> an inherent bug. For the time being, I created a copyfield which uses a
> custom regex to remove whitespace and special characters and use it in the
> function. I'll debug the source code and confirm if it's bug, will raise a
> JIRA if needed.
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Reply | Threaded
Open this post in threaded view
|

Re: Problem with white space or special characters in function queries

shamik
Thanks Jan, I was not aware of this, appreciate your help.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Problem with white space or special characters in function queries

Yonik Seeley
In reply to this post by Jan Høydahl / Cominvent
On Thu, Mar 28, 2019 at 6:05 PM Jan Høydahl <[hidden email]> wrote:

> Functions can never contain spaces.


Spaces work fine in functions in general.
The issue is the "bf" parameter as it uses whitespace to delimit multiple
functions IIRC.

-Yonik



> Try to substitute the term with a variable, i.e. a request parameter, e.g.
>
>
> bf=if(termfreq(ADSKFeature,$myTerm),log(CaseCount),sqrt(CaseCount))&myTerm=CUI+(Command)
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 28. mar. 2019 kl. 18:51 skrev shamik <[hidden email]>:
> >
> > Ahemad, I don't think its related to the field definition, rather looks
> like
> > an inherent bug. For the time being, I created a copyfield which uses a
> > custom regex to remove whitespace and special characters and use it in
> the
> > function. I'll debug the source code and confirm if it's bug, will raise
> a
> > JIRA if needed.
> >
> >
> >
> > --
> > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Problem with white space or special characters in function queries

Ahemad Ali
Do you have any suggestions for querying the indexed data with whitespaces and special charectors ?

Sent from Yahoo Mail on Android
 
  On Fri, Mar 29, 2019 at 16:59, Yonik Seeley<[hidden email]> wrote:   On Thu, Mar 28, 2019 at 6:05 PM Jan Høydahl <[hidden email]> wrote:

> Functions can never contain spaces.


Spaces work fine in functions in general.
The issue is the "bf" parameter as it uses whitespace to delimit multiple
functions IIRC.

-Yonik



> Try to substitute the term with a variable, i.e. a request parameter, e.g.
>
>
> bf=if(termfreq(ADSKFeature,$myTerm),log(CaseCount),sqrt(CaseCount))&myTerm=CUI+(Command)
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 28. mar. 2019 kl. 18:51 skrev shamik <[hidden email]>:
> >
> > Ahemad, I don't think its related to the field definition, rather looks
> like
> > an inherent bug. For the time being, I created a copyfield which uses a
> > custom regex to remove whitespace and special characters and use it in
> the
> > function. I'll debug the source code and confirm if it's bug, will raise
> a
> > JIRA if needed.
> >
> >
> >
> > --
> > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
>  
Reply | Threaded
Open this post in threaded view
|

Re: Problem with white space or special characters in function queries

Erick Erickson
Ahamed:

Please start a new thread. Although your question is somewhat related, it’s not the same thing at all. It’s called “thread hijacking” and makes it difficult to keep track of.

> On Mar 29, 2019, at 4:46 AM, Ahemad Ali <[hidden email]> wrote:
>
> Do you have any suggestions for querying the indexed data with whitespaces and special charectors ?
>
> Sent from Yahoo Mail on Android
>
>  On Fri, Mar 29, 2019 at 16:59, Yonik Seeley<[hidden email]> wrote:   On Thu, Mar 28, 2019 at 6:05 PM Jan Høydahl <[hidden email]> wrote:
>
>> Functions can never contain spaces.
>
>
> Spaces work fine in functions in general.
> The issue is the "bf" parameter as it uses whitespace to delimit multiple
> functions IIRC.
>
> -Yonik
>
>
>
>> Try to substitute the term with a variable, i.e. a request parameter, e.g.
>>
>>
>> bf=if(termfreq(ADSKFeature,$myTerm),log(CaseCount),sqrt(CaseCount))&myTerm=CUI+(Command)
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>>> 28. mar. 2019 kl. 18:51 skrev shamik <[hidden email]>:
>>>
>>> Ahemad, I don't think its related to the field definition, rather looks
>> like
>>> an inherent bug. For the time being, I created a copyfield which uses a
>>> custom regex to remove whitespace and special characters and use it in
>> the
>>> function. I'll debug the source code and confirm if it's bug, will raise
>> a
>>> JIRA if needed.
>>>
>>>
>>>
>>> --
>>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>>
>>