Solr shingles is not working in solr 6.4.0

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr shingles is not working in solr 6.4.0

Aman Deep Singh
Hi,

Recently I migrated from solr 4 to 6
IN solr 4 shinglefilterfactory is working correctly
my configration  i

    <fieldType name="text_shingle" class="solr.TextField"
positionIncrementGap="100">
        <analyzer type="index">
         <tokenizer class="solr.WhitespaceTokenizerFactory"/>
         <filter class="solr.ShingleFilterFactory" minShingleSize="2"
maxShingleSize="5"
                 outputUnigrams="false" outputUnigramsIfNoShingles="false" />
          <filter class="solr.LowerCaseFilterFactory"/>
        </analyzer>
        <analyzer type="query">
          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
         <filter class="solr.ShingleFilterFactory" minShingleSize="2"
maxShingleSize="5"
                 outputUnigrams="false" outputUnigramsIfNoShingles="false" />
          <filter class="solr.LowerCaseFilterFactory"/>
          <filter class="solr.PositionFilterFactory"/>
        </analyzer>
      </fieldType>



But after updating to solr 6 shingles is not working ,schema is as below,

        <fieldType name="text_shingle" class="solr.TextField"
positionIncrementGap="100">
        <analyzer type="index">
         <tokenizer class="solr.WhitespaceTokenizerFactory"/>
         <filter class="solr.ShingleFilterFactory" minShingleSize="2"
maxShingleSize="5"
                 outputUnigrams="false" outputUnigramsIfNoShingles="false" />
          <filter class="solr.LowerCaseFilterFactory"/>
        </analyzer>
        <analyzer type="query">
          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
         <filter class="solr.ShingleFilterFactory" minShingleSize="2"
maxShingleSize="5"
                 outputUnigrams="false" outputUnigramsIfNoShingles="false" />
          <filter class="solr.LowerCaseFilterFactory"/>
        </analyzer>
      </fieldType>

Although in the Analysis tab is was showing proper shingle result but when
using in the queryParser it was not giving proper results

my sample hit is

http://localhost:8983/solr/shingel_test/select?q=one%20plus%20one&wt=xml&debugQuery=true&defType=edismax&qf=cust_shingle

it create the parsed query as

<str name="rawquerystring">one plus one</str>
<str name="querystring">one plus one</str>
<str name="parsedquery">(+())/no_coord</str>
<str name="parsedquery_toString">+()</str>
<lst name="explain"/>
<str name="QParser">ExtendedDismaxQParser</str>
Reply | Threaded
Open this post in threaded view
|

Re: Solr shingles is not working in solr 6.4.0

alessandro.benedetti
Hi Aman, are you using stopword in your analysis by any chance ?
Can you show us your request handler config ?
With the edismax you can configure stopwords to take effect at query parsing stage.
Let's try to figure it out first.

Cheers
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
Reply | Threaded
Open this post in threaded view
|

Re: Solr shingles is not working in solr 6.4.0

Alexandre Rafalovitch
In reply to this post by Aman Deep Singh
Sanity check. Is your 'df' pointing at the field you think it is
pointing at? It really does look like all tokens were eaten and
nothing was left. But you should have seen that in the Analysis screen
too, if you have the right field.

Try adding echoParams=all to your request to see the full final
parameter list. Maybe some parameters in initParams sections override
your assumed config.

Regards,
   Alex.
----
http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 16 March 2017 at 08:30, Aman Deep Singh <[hidden email]> wrote:

> Hi,
>
> Recently I migrated from solr 4 to 6
> IN solr 4 shinglefilterfactory is working correctly
> my configration  i
>
>     <fieldType name="text_shingle" class="solr.TextField"
> positionIncrementGap="100">
>         <analyzer type="index">
>          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
> maxShingleSize="5"
>                  outputUnigrams="false" outputUnigramsIfNoShingles="false" />
>           <filter class="solr.LowerCaseFilterFactory"/>
>         </analyzer>
>         <analyzer type="query">
>           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
> maxShingleSize="5"
>                  outputUnigrams="false" outputUnigramsIfNoShingles="false" />
>           <filter class="solr.LowerCaseFilterFactory"/>
>           <filter class="solr.PositionFilterFactory"/>
>         </analyzer>
>       </fieldType>
>
>
>
> But after updating to solr 6 shingles is not working ,schema is as below,
>
>         <fieldType name="text_shingle" class="solr.TextField"
> positionIncrementGap="100">
>         <analyzer type="index">
>          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
> maxShingleSize="5"
>                  outputUnigrams="false" outputUnigramsIfNoShingles="false" />
>           <filter class="solr.LowerCaseFilterFactory"/>
>         </analyzer>
>         <analyzer type="query">
>           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
> maxShingleSize="5"
>                  outputUnigrams="false" outputUnigramsIfNoShingles="false" />
>           <filter class="solr.LowerCaseFilterFactory"/>
>         </analyzer>
>       </fieldType>
>
> Although in the Analysis tab is was showing proper shingle result but when
> using in the queryParser it was not giving proper results
>
> my sample hit is
>
> http://localhost:8983/solr/shingel_test/select?q=one%20plus%20one&wt=xml&debugQuery=true&defType=edismax&qf=cust_shingle
>
> it create the parsed query as
>
> <str name="rawquerystring">one plus one</str>
> <str name="querystring">one plus one</str>
> <str name="parsedquery">(+())/no_coord</str>
> <str name="parsedquery_toString">+()</str>
> <lst name="explain"/>
> <str name="QParser">ExtendedDismaxQParser</str>
Reply | Threaded
Open this post in threaded view
|

Re: Solr shingles is not working in solr 6.4.0

Aman Deep Singh
Already check that i am sending sceenshots of various senarios  

On Thu, Mar 16, 2017 at 7:46 PM Alexandre Rafalovitch <[hidden email]> wrote:
Sanity check. Is your 'df' pointing at the field you think it is
pointing at? It really does look like all tokens were eaten and
nothing was left. But you should have seen that in the Analysis screen
too, if you have the right field.

Try adding echoParams=all to your request to see the full final
parameter list. Maybe some parameters in initParams sections override
your assumed config.

Regards,
   Alex.
----
http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 16 March 2017 at 08:30, Aman Deep Singh <[hidden email]> wrote:
> Hi,
>
> Recently I migrated from solr 4 to 6
> IN solr 4 shinglefilterfactory is working correctly
> my configration  i
>
>     <fieldType name="text_shingle" class="solr.TextField"
> positionIncrementGap="100">
>         <analyzer type="index">
>          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
> maxShingleSize="5"
>                  outputUnigrams="false" outputUnigramsIfNoShingles="false" />
>           <filter class="solr.LowerCaseFilterFactory"/>
>         </analyzer>
>         <analyzer type="query">
>           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
> maxShingleSize="5"
>                  outputUnigrams="false" outputUnigramsIfNoShingles="false" />
>           <filter class="solr.LowerCaseFilterFactory"/>
>           <filter class="solr.PositionFilterFactory"/>
>         </analyzer>
>       </fieldType>
>
>
>
> But after updating to solr 6 shingles is not working ,schema is as below,
>
>         <fieldType name="text_shingle" class="solr.TextField"
> positionIncrementGap="100">
>         <analyzer type="index">
>          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
> maxShingleSize="5"
>                  outputUnigrams="false" outputUnigramsIfNoShingles="false" />
>           <filter class="solr.LowerCaseFilterFactory"/>
>         </analyzer>
>         <analyzer type="query">
>           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
> maxShingleSize="5"
>                  outputUnigrams="false" outputUnigramsIfNoShingles="false" />
>           <filter class="solr.LowerCaseFilterFactory"/>
>         </analyzer>
>       </fieldType>
>
> Although in the Analysis tab is was showing proper shingle result but when
> using in the queryParser it was not giving proper results
>
> my sample hit is
>
> http://localhost:8983/solr/shingel_test/select?q=one%20plus%20one&wt=xml&debugQuery=true&defType=edismax&qf=cust_shingle
>
> it create the parsed query as
>
> <str name="rawquerystring">one plus one</str>
> <str name="querystring">one plus one</str>
> <str name="parsedquery">(+())/no_coord</str>
> <str name="parsedquery_toString">+()</str>
> <lst name="explain"/>
> <str name="QParser">ExtendedDismaxQParser</str>
Reply | Threaded
Open this post in threaded view
|

Re: Solr shingles is not working in solr 6.4.0

Alexandre Rafalovitch
Images do not come through.

But I was wrong too. You use eDismax and pass "cust_shingle" in, so
the "df" value is irrelevant.

You definitely reloaded the core after changing definitions?
----
http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 16 March 2017 at 12:37, Aman Deep Singh <[hidden email]> wrote:

> Already check that i am sending sceenshots of various senarios
>
>
> On Thu, Mar 16, 2017 at 7:46 PM Alexandre Rafalovitch <[hidden email]>
> wrote:
>>
>> Sanity check. Is your 'df' pointing at the field you think it is
>> pointing at? It really does look like all tokens were eaten and
>> nothing was left. But you should have seen that in the Analysis screen
>> too, if you have the right field.
>>
>> Try adding echoParams=all to your request to see the full final
>> parameter list. Maybe some parameters in initParams sections override
>> your assumed config.
>>
>> Regards,
>>    Alex.
>> ----
>> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>>
>>
>> On 16 March 2017 at 08:30, Aman Deep Singh <[hidden email]>
>> wrote:
>> > Hi,
>> >
>> > Recently I migrated from solr 4 to 6
>> > IN solr 4 shinglefilterfactory is working correctly
>> > my configration  i
>> >
>> >     <fieldType name="text_shingle" class="solr.TextField"
>> > positionIncrementGap="100">
>> >         <analyzer type="index">
>> >          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> >          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
>> > maxShingleSize="5"
>> >                  outputUnigrams="false"
>> > outputUnigramsIfNoShingles="false" />
>> >           <filter class="solr.LowerCaseFilterFactory"/>
>> >         </analyzer>
>> >         <analyzer type="query">
>> >           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> >          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
>> > maxShingleSize="5"
>> >                  outputUnigrams="false"
>> > outputUnigramsIfNoShingles="false" />
>> >           <filter class="solr.LowerCaseFilterFactory"/>
>> >           <filter class="solr.PositionFilterFactory"/>
>> >         </analyzer>
>> >       </fieldType>
>> >
>> >
>> >
>> > But after updating to solr 6 shingles is not working ,schema is as
>> > below,
>> >
>> >         <fieldType name="text_shingle" class="solr.TextField"
>> > positionIncrementGap="100">
>> >         <analyzer type="index">
>> >          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> >          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
>> > maxShingleSize="5"
>> >                  outputUnigrams="false"
>> > outputUnigramsIfNoShingles="false" />
>> >           <filter class="solr.LowerCaseFilterFactory"/>
>> >         </analyzer>
>> >         <analyzer type="query">
>> >           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> >          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
>> > maxShingleSize="5"
>> >                  outputUnigrams="false"
>> > outputUnigramsIfNoShingles="false" />
>> >           <filter class="solr.LowerCaseFilterFactory"/>
>> >         </analyzer>
>> >       </fieldType>
>> >
>> > Although in the Analysis tab is was showing proper shingle result but
>> > when
>> > using in the queryParser it was not giving proper results
>> >
>> > my sample hit is
>> >
>> >
>> > http://localhost:8983/solr/shingel_test/select?q=one%20plus%20one&wt=xml&debugQuery=true&defType=edismax&qf=cust_shingle
>> >
>> > it create the parsed query as
>> >
>> > <str name="rawquerystring">one plus one</str>
>> > <str name="querystring">one plus one</str>
>> > <str name="parsedquery">(+())/no_coord</str>
>> > <str name="parsedquery_toString">+()</str>
>> > <lst name="explain"/>
>> > <str name="QParser">ExtendedDismaxQParser</str>
Reply | Threaded
Open this post in threaded view
|

Re: Solr shingles is not working in solr 6.4.0

Aman Deep Singh
Yes I have reloaded the core after config changes

On 16-Mar-2017 10:28 PM, "Alexandre Rafalovitch" <[hidden email]> wrote:

Images do not come through.

But I was wrong too. You use eDismax and pass "cust_shingle" in, so
the "df" value is irrelevant.

You definitely reloaded the core after changing definitions?
----
http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 16 March 2017 at 12:37, Aman Deep Singh <[hidden email]>
wrote:

> Already check that i am sending sceenshots of various senarios
>
>
> On Thu, Mar 16, 2017 at 7:46 PM Alexandre Rafalovitch <[hidden email]>
> wrote:
>>
>> Sanity check. Is your 'df' pointing at the field you think it is
>> pointing at? It really does look like all tokens were eaten and
>> nothing was left. But you should have seen that in the Analysis screen
>> too, if you have the right field.
>>
>> Try adding echoParams=all to your request to see the full final
>> parameter list. Maybe some parameters in initParams sections override
>> your assumed config.
>>
>> Regards,
>>    Alex.
>> ----
>> http://www.solr-start.com/ - Resources for Solr users, new and
experienced

>>
>>
>> On 16 March 2017 at 08:30, Aman Deep Singh <[hidden email]>
>> wrote:
>> > Hi,
>> >
>> > Recently I migrated from solr 4 to 6
>> > IN solr 4 shinglefilterfactory is working correctly
>> > my configration  i
>> >
>> >     <fieldType name="text_shingle" class="solr.TextField"
>> > positionIncrementGap="100">
>> >         <analyzer type="index">
>> >          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> >          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
>> > maxShingleSize="5"
>> >                  outputUnigrams="false"
>> > outputUnigramsIfNoShingles="false" />
>> >           <filter class="solr.LowerCaseFilterFactory"/>
>> >         </analyzer>
>> >         <analyzer type="query">
>> >           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> >          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
>> > maxShingleSize="5"
>> >                  outputUnigrams="false"
>> > outputUnigramsIfNoShingles="false" />
>> >           <filter class="solr.LowerCaseFilterFactory"/>
>> >           <filter class="solr.PositionFilterFactory"/>
>> >         </analyzer>
>> >       </fieldType>
>> >
>> >
>> >
>> > But after updating to solr 6 shingles is not working ,schema is as
>> > below,
>> >
>> >         <fieldType name="text_shingle" class="solr.TextField"
>> > positionIncrementGap="100">
>> >         <analyzer type="index">
>> >          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> >          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
>> > maxShingleSize="5"
>> >                  outputUnigrams="false"
>> > outputUnigramsIfNoShingles="false" />
>> >           <filter class="solr.LowerCaseFilterFactory"/>
>> >         </analyzer>
>> >         <analyzer type="query">
>> >           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> >          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
>> > maxShingleSize="5"
>> >                  outputUnigrams="false"
>> > outputUnigramsIfNoShingles="false" />
>> >           <filter class="solr.LowerCaseFilterFactory"/>
>> >         </analyzer>
>> >       </fieldType>
>> >
>> > Although in the Analysis tab is was showing proper shingle result but
>> > when
>> > using in the queryParser it was not giving proper results
>> >
>> > my sample hit is
>> >
>> >
>> > <a href="http://localhost:8983/solr/shingel_test/select?q=one%">http://localhost:8983/solr/shingel_test/select?q=one%
20plus%20one&wt=xml&debugQuery=true&defType=edismax&qf=cust_shingle
>> >
>> > it create the parsed query as
>> >
>> > <str name="rawquerystring">one plus one</str>
>> > <str name="querystring">one plus one</str>
>> > <str name="parsedquery">(+())/no_coord</str>
>> > <str name="parsedquery_toString">+()</str>
>> > <lst name="explain"/>
>> > <str name="QParser">ExtendedDismaxQParser</str>
Reply | Threaded
Open this post in threaded view
|

Re: Solr shingles is not working in solr 6.4.0

Aman Deep Singh
For images dropbox url is
https://www.dropbox.com/sh/6dy6a8ajabjtxrt/AAAoxhZQe2vp3sTl3Av71_eHa?dl=0


On Thu, Mar 16, 2017 at 10:29 PM Aman Deep Singh <[hidden email]>
wrote:

> Yes I have reloaded the core after config changes
>
>
> On 16-Mar-2017 10:28 PM, "Alexandre Rafalovitch" <[hidden email]>
> wrote:
>
> Images do not come through.
>
> But I was wrong too. You use eDismax and pass "cust_shingle" in, so
> the "df" value is irrelevant.
>
> You definitely reloaded the core after changing definitions?
> ----
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 16 March 2017 at 12:37, Aman Deep Singh <[hidden email]>
> wrote:
> > Already check that i am sending sceenshots of various senarios
> >
> >
> > On Thu, Mar 16, 2017 at 7:46 PM Alexandre Rafalovitch <
> [hidden email]>
> > wrote:
> >>
> >> Sanity check. Is your 'df' pointing at the field you think it is
> >> pointing at? It really does look like all tokens were eaten and
> >> nothing was left. But you should have seen that in the Analysis screen
> >> too, if you have the right field.
> >>
> >> Try adding echoParams=all to your request to see the full final
> >> parameter list. Maybe some parameters in initParams sections override
> >> your assumed config.
> >>
> >> Regards,
> >>    Alex.
> >> ----
> >> http://www.solr-start.com/ - Resources for Solr users, new and
> experienced
> >>
> >>
> >> On 16 March 2017 at 08:30, Aman Deep Singh <[hidden email]>
> >> wrote:
> >> > Hi,
> >> >
> >> > Recently I migrated from solr 4 to 6
> >> > IN solr 4 shinglefilterfactory is working correctly
> >> > my configration  i
> >> >
> >> >     <fieldType name="text_shingle" class="solr.TextField"
> >> > positionIncrementGap="100">
> >> >         <analyzer type="index">
> >> >          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> >> >          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
> >> > maxShingleSize="5"
> >> >                  outputUnigrams="false"
> >> > outputUnigramsIfNoShingles="false" />
> >> >           <filter class="solr.LowerCaseFilterFactory"/>
> >> >         </analyzer>
> >> >         <analyzer type="query">
> >> >           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> >> >          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
> >> > maxShingleSize="5"
> >> >                  outputUnigrams="false"
> >> > outputUnigramsIfNoShingles="false" />
> >> >           <filter class="solr.LowerCaseFilterFactory"/>
> >> >           <filter class="solr.PositionFilterFactory"/>
> >> >         </analyzer>
> >> >       </fieldType>
> >> >
> >> >
> >> >
> >> > But after updating to solr 6 shingles is not working ,schema is as
> >> > below,
> >> >
> >> >         <fieldType name="text_shingle" class="solr.TextField"
> >> > positionIncrementGap="100">
> >> >         <analyzer type="index">
> >> >          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> >> >          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
> >> > maxShingleSize="5"
> >> >                  outputUnigrams="false"
> >> > outputUnigramsIfNoShingles="false" />
> >> >           <filter class="solr.LowerCaseFilterFactory"/>
> >> >         </analyzer>
> >> >         <analyzer type="query">
> >> >           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> >> >          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
> >> > maxShingleSize="5"
> >> >                  outputUnigrams="false"
> >> > outputUnigramsIfNoShingles="false" />
> >> >           <filter class="solr.LowerCaseFilterFactory"/>
> >> >         </analyzer>
> >> >       </fieldType>
> >> >
> >> > Although in the Analysis tab is was showing proper shingle result but
> >> > when
> >> > using in the queryParser it was not giving proper results
> >> >
> >> > my sample hit is
> >> >
> >> >
> >> >
> http://localhost:8983/solr/shingel_test/select?q=one%20plus%20one&wt=xml&debugQuery=true&defType=edismax&qf=cust_shingle
> >> >
> >> > it create the parsed query as
> >> >
> >> > <str name="rawquerystring">one plus one</str>
> >> > <str name="querystring">one plus one</str>
> >> > <str name="parsedquery">(+())/no_coord</str>
> >> > <str name="parsedquery_toString">+()</str>
> >> > <lst name="explain"/>
> >> > <str name="QParser">ExtendedDismaxQParser</str>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr shingles is not working in solr 6.4.0

Alexandre Rafalovitch
Oh. Try your query with quotes around the phone phrase:
q="one plus one"

My hypothesis is:
Query parser splits things on whitespace before passing it down into
analyzer chain as individual match attempts. The Analysis UI does not
take that into account and treats the whole string as phrase sent. You
say
outputUnigrams="false" outputUnigramsIfNoShingles="false"
So, every single token during the query gets ignored because there is
nothing for it to shingle with.

I am not sure why it would have worked in Solr 4.

Regards,
   Alex.
----
http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 16 March 2017 at 13:06, Aman Deep Singh <[hidden email]> wrote:

> For images dropbox url is
> https://www.dropbox.com/sh/6dy6a8ajabjtxrt/AAAoxhZQe2vp3sTl3Av71_eHa?dl=0
>
>
> On Thu, Mar 16, 2017 at 10:29 PM Aman Deep Singh <[hidden email]>
> wrote:
>
>> Yes I have reloaded the core after config changes
>>
>>
>> On 16-Mar-2017 10:28 PM, "Alexandre Rafalovitch" <[hidden email]>
>> wrote:
>>
>> Images do not come through.
>>
>> But I was wrong too. You use eDismax and pass "cust_shingle" in, so
>> the "df" value is irrelevant.
>>
>> You definitely reloaded the core after changing definitions?
>> ----
>> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>>
>>
>> On 16 March 2017 at 12:37, Aman Deep Singh <[hidden email]>
>> wrote:
>> > Already check that i am sending sceenshots of various senarios
>> >
>> >
>> > On Thu, Mar 16, 2017 at 7:46 PM Alexandre Rafalovitch <
>> [hidden email]>
>> > wrote:
>> >>
>> >> Sanity check. Is your 'df' pointing at the field you think it is
>> >> pointing at? It really does look like all tokens were eaten and
>> >> nothing was left. But you should have seen that in the Analysis screen
>> >> too, if you have the right field.
>> >>
>> >> Try adding echoParams=all to your request to see the full final
>> >> parameter list. Maybe some parameters in initParams sections override
>> >> your assumed config.
>> >>
>> >> Regards,
>> >>    Alex.
>> >> ----
>> >> http://www.solr-start.com/ - Resources for Solr users, new and
>> experienced
>> >>
>> >>
>> >> On 16 March 2017 at 08:30, Aman Deep Singh <[hidden email]>
>> >> wrote:
>> >> > Hi,
>> >> >
>> >> > Recently I migrated from solr 4 to 6
>> >> > IN solr 4 shinglefilterfactory is working correctly
>> >> > my configration  i
>> >> >
>> >> >     <fieldType name="text_shingle" class="solr.TextField"
>> >> > positionIncrementGap="100">
>> >> >         <analyzer type="index">
>> >> >          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> >> >          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
>> >> > maxShingleSize="5"
>> >> >                  outputUnigrams="false"
>> >> > outputUnigramsIfNoShingles="false" />
>> >> >           <filter class="solr.LowerCaseFilterFactory"/>
>> >> >         </analyzer>
>> >> >         <analyzer type="query">
>> >> >           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> >> >          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
>> >> > maxShingleSize="5"
>> >> >                  outputUnigrams="false"
>> >> > outputUnigramsIfNoShingles="false" />
>> >> >           <filter class="solr.LowerCaseFilterFactory"/>
>> >> >           <filter class="solr.PositionFilterFactory"/>
>> >> >         </analyzer>
>> >> >       </fieldType>
>> >> >
>> >> >
>> >> >
>> >> > But after updating to solr 6 shingles is not working ,schema is as
>> >> > below,
>> >> >
>> >> >         <fieldType name="text_shingle" class="solr.TextField"
>> >> > positionIncrementGap="100">
>> >> >         <analyzer type="index">
>> >> >          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> >> >          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
>> >> > maxShingleSize="5"
>> >> >                  outputUnigrams="false"
>> >> > outputUnigramsIfNoShingles="false" />
>> >> >           <filter class="solr.LowerCaseFilterFactory"/>
>> >> >         </analyzer>
>> >> >         <analyzer type="query">
>> >> >           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> >> >          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
>> >> > maxShingleSize="5"
>> >> >                  outputUnigrams="false"
>> >> > outputUnigramsIfNoShingles="false" />
>> >> >           <filter class="solr.LowerCaseFilterFactory"/>
>> >> >         </analyzer>
>> >> >       </fieldType>
>> >> >
>> >> > Although in the Analysis tab is was showing proper shingle result but
>> >> > when
>> >> > using in the queryParser it was not giving proper results
>> >> >
>> >> > my sample hit is
>> >> >
>> >> >
>> >> >
>> http://localhost:8983/solr/shingel_test/select?q=one%20plus%20one&wt=xml&debugQuery=true&defType=edismax&qf=cust_shingle
>> >> >
>> >> > it create the parsed query as
>> >> >
>> >> > <str name="rawquerystring">one plus one</str>
>> >> > <str name="querystring">one plus one</str>
>> >> > <str name="parsedquery">(+())/no_coord</str>
>> >> > <str name="parsedquery_toString">+()</str>
>> >> > <lst name="explain"/>
>> >> > <str name="QParser">ExtendedDismaxQParser</str>
>>
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: Solr shingles is not working in solr 6.4.0

Aman Deep Singh
If I give query in quotes it converted query in to graph query as

Graph(cust_sh6:"one plus one" hasBoolean=false hasPhrase=false)


On 17-Mar-2017 1:38 AM, "Alexandre Rafalovitch" <[hidden email]> wrote:

> Oh. Try your query with quotes around the phone phrase:
> q="one plus one"
>
> My hypothesis is:
> Query parser splits things on whitespace before passing it down into
> analyzer chain as individual match attempts. The Analysis UI does not
> take that into account and treats the whole string as phrase sent. You
> say
> outputUnigrams="false" outputUnigramsIfNoShingles="false"
> So, every single token during the query gets ignored because there is
> nothing for it to shingle with.
>
> I am not sure why it would have worked in Solr 4.
>
> Regards,
>    Alex.
> ----
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 16 March 2017 at 13:06, Aman Deep Singh <[hidden email]>
> wrote:
> > For images dropbox url is
> > https://www.dropbox.com/sh/6dy6a8ajabjtxrt/
> AAAoxhZQe2vp3sTl3Av71_eHa?dl=0
> >
> >
> > On Thu, Mar 16, 2017 at 10:29 PM Aman Deep Singh <
> [hidden email]>
> > wrote:
> >
> >> Yes I have reloaded the core after config changes
> >>
> >>
> >> On 16-Mar-2017 10:28 PM, "Alexandre Rafalovitch" <[hidden email]>
> >> wrote:
> >>
> >> Images do not come through.
> >>
> >> But I was wrong too. You use eDismax and pass "cust_shingle" in, so
> >> the "df" value is irrelevant.
> >>
> >> You definitely reloaded the core after changing definitions?
> >> ----
> >> http://www.solr-start.com/ - Resources for Solr users, new and
> experienced
> >>
> >>
> >> On 16 March 2017 at 12:37, Aman Deep Singh <[hidden email]>
> >> wrote:
> >> > Already check that i am sending sceenshots of various senarios
> >> >
> >> >
> >> > On Thu, Mar 16, 2017 at 7:46 PM Alexandre Rafalovitch <
> >> [hidden email]>
> >> > wrote:
> >> >>
> >> >> Sanity check. Is your 'df' pointing at the field you think it is
> >> >> pointing at? It really does look like all tokens were eaten and
> >> >> nothing was left. But you should have seen that in the Analysis
> screen
> >> >> too, if you have the right field.
> >> >>
> >> >> Try adding echoParams=all to your request to see the full final
> >> >> parameter list. Maybe some parameters in initParams sections override
> >> >> your assumed config.
> >> >>
> >> >> Regards,
> >> >>    Alex.
> >> >> ----
> >> >> http://www.solr-start.com/ - Resources for Solr users, new and
> >> experienced
> >> >>
> >> >>
> >> >> On 16 March 2017 at 08:30, Aman Deep Singh <
> [hidden email]>
> >> >> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > Recently I migrated from solr 4 to 6
> >> >> > IN solr 4 shinglefilterfactory is working correctly
> >> >> > my configration  i
> >> >> >
> >> >> >     <fieldType name="text_shingle" class="solr.TextField"
> >> >> > positionIncrementGap="100">
> >> >> >         <analyzer type="index">
> >> >> >          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> >> >> >          <filter class="solr.ShingleFilterFactory"
> minShingleSize="2"
> >> >> > maxShingleSize="5"
> >> >> >                  outputUnigrams="false"
> >> >> > outputUnigramsIfNoShingles="false" />
> >> >> >           <filter class="solr.LowerCaseFilterFactory"/>
> >> >> >         </analyzer>
> >> >> >         <analyzer type="query">
> >> >> >           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> >> >> >          <filter class="solr.ShingleFilterFactory"
> minShingleSize="2"
> >> >> > maxShingleSize="5"
> >> >> >                  outputUnigrams="false"
> >> >> > outputUnigramsIfNoShingles="false" />
> >> >> >           <filter class="solr.LowerCaseFilterFactory"/>
> >> >> >           <filter class="solr.PositionFilterFactory"/>
> >> >> >         </analyzer>
> >> >> >       </fieldType>
> >> >> >
> >> >> >
> >> >> >
> >> >> > But after updating to solr 6 shingles is not working ,schema is as
> >> >> > below,
> >> >> >
> >> >> >         <fieldType name="text_shingle" class="solr.TextField"
> >> >> > positionIncrementGap="100">
> >> >> >         <analyzer type="index">
> >> >> >          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> >> >> >          <filter class="solr.ShingleFilterFactory"
> minShingleSize="2"
> >> >> > maxShingleSize="5"
> >> >> >                  outputUnigrams="false"
> >> >> > outputUnigramsIfNoShingles="false" />
> >> >> >           <filter class="solr.LowerCaseFilterFactory"/>
> >> >> >         </analyzer>
> >> >> >         <analyzer type="query">
> >> >> >           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> >> >> >          <filter class="solr.ShingleFilterFactory"
> minShingleSize="2"
> >> >> > maxShingleSize="5"
> >> >> >                  outputUnigrams="false"
> >> >> > outputUnigramsIfNoShingles="false" />
> >> >> >           <filter class="solr.LowerCaseFilterFactory"/>
> >> >> >         </analyzer>
> >> >> >       </fieldType>
> >> >> >
> >> >> > Although in the Analysis tab is was showing proper shingle result
> but
> >> >> > when
> >> >> > using in the queryParser it was not giving proper results
> >> >> >
> >> >> > my sample hit is
> >> >> >
> >> >> >
> >> >> >
> >> <a href="http://localhost:8983/solr/shingel_test/select?q=one%">http://localhost:8983/solr/shingel_test/select?q=one%
> 20plus%20one&wt=xml&debugQuery=true&defType=edismax&qf=cust_shingle
> >> >> >
> >> >> > it create the parsed query as
> >> >> >
> >> >> > <str name="rawquerystring">one plus one</str>
> >> >> > <str name="querystring">one plus one</str>
> >> >> > <str name="parsedquery">(+())/no_coord</str>
> >> >> > <str name="parsedquery_toString">+()</str>
> >> >> > <lst name="explain"/>
> >> >> > <str name="QParser">ExtendedDismaxQParser</str>
> >>
> >>
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr shingles is not working in solr 6.4.0

Alexandre Rafalovitch
Which is what I believe you had as a working example in your Dropbox images.

So, does it work now?

Regards,
   Alex.
----
http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 16 March 2017 at 22:22, Aman Deep Singh <[hidden email]> wrote:

> If I give query in quotes it converted query in to graph query as
>
> Graph(cust_sh6:"one plus one" hasBoolean=false hasPhrase=false)
>
>
> On 17-Mar-2017 1:38 AM, "Alexandre Rafalovitch" <[hidden email]> wrote:
>
>> Oh. Try your query with quotes around the phone phrase:
>> q="one plus one"
>>
>> My hypothesis is:
>> Query parser splits things on whitespace before passing it down into
>> analyzer chain as individual match attempts. The Analysis UI does not
>> take that into account and treats the whole string as phrase sent. You
>> say
>> outputUnigrams="false" outputUnigramsIfNoShingles="false"
>> So, every single token during the query gets ignored because there is
>> nothing for it to shingle with.
>>
>> I am not sure why it would have worked in Solr 4.
>>
>> Regards,
>>    Alex.
>> ----
>> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>>
>>
>> On 16 March 2017 at 13:06, Aman Deep Singh <[hidden email]>
>> wrote:
>> > For images dropbox url is
>> > https://www.dropbox.com/sh/6dy6a8ajabjtxrt/
>> AAAoxhZQe2vp3sTl3Av71_eHa?dl=0
>> >
>> >
>> > On Thu, Mar 16, 2017 at 10:29 PM Aman Deep Singh <
>> [hidden email]>
>> > wrote:
>> >
>> >> Yes I have reloaded the core after config changes
>> >>
>> >>
>> >> On 16-Mar-2017 10:28 PM, "Alexandre Rafalovitch" <[hidden email]>
>> >> wrote:
>> >>
>> >> Images do not come through.
>> >>
>> >> But I was wrong too. You use eDismax and pass "cust_shingle" in, so
>> >> the "df" value is irrelevant.
>> >>
>> >> You definitely reloaded the core after changing definitions?
>> >> ----
>> >> http://www.solr-start.com/ - Resources for Solr users, new and
>> experienced
>> >>
>> >>
>> >> On 16 March 2017 at 12:37, Aman Deep Singh <[hidden email]>
>> >> wrote:
>> >> > Already check that i am sending sceenshots of various senarios
>> >> >
>> >> >
>> >> > On Thu, Mar 16, 2017 at 7:46 PM Alexandre Rafalovitch <
>> >> [hidden email]>
>> >> > wrote:
>> >> >>
>> >> >> Sanity check. Is your 'df' pointing at the field you think it is
>> >> >> pointing at? It really does look like all tokens were eaten and
>> >> >> nothing was left. But you should have seen that in the Analysis
>> screen
>> >> >> too, if you have the right field.
>> >> >>
>> >> >> Try adding echoParams=all to your request to see the full final
>> >> >> parameter list. Maybe some parameters in initParams sections override
>> >> >> your assumed config.
>> >> >>
>> >> >> Regards,
>> >> >>    Alex.
>> >> >> ----
>> >> >> http://www.solr-start.com/ - Resources for Solr users, new and
>> >> experienced
>> >> >>
>> >> >>
>> >> >> On 16 March 2017 at 08:30, Aman Deep Singh <
>> [hidden email]>
>> >> >> wrote:
>> >> >> > Hi,
>> >> >> >
>> >> >> > Recently I migrated from solr 4 to 6
>> >> >> > IN solr 4 shinglefilterfactory is working correctly
>> >> >> > my configration  i
>> >> >> >
>> >> >> >     <fieldType name="text_shingle" class="solr.TextField"
>> >> >> > positionIncrementGap="100">
>> >> >> >         <analyzer type="index">
>> >> >> >          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> >> >> >          <filter class="solr.ShingleFilterFactory"
>> minShingleSize="2"
>> >> >> > maxShingleSize="5"
>> >> >> >                  outputUnigrams="false"
>> >> >> > outputUnigramsIfNoShingles="false" />
>> >> >> >           <filter class="solr.LowerCaseFilterFactory"/>
>> >> >> >         </analyzer>
>> >> >> >         <analyzer type="query">
>> >> >> >           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> >> >> >          <filter class="solr.ShingleFilterFactory"
>> minShingleSize="2"
>> >> >> > maxShingleSize="5"
>> >> >> >                  outputUnigrams="false"
>> >> >> > outputUnigramsIfNoShingles="false" />
>> >> >> >           <filter class="solr.LowerCaseFilterFactory"/>
>> >> >> >           <filter class="solr.PositionFilterFactory"/>
>> >> >> >         </analyzer>
>> >> >> >       </fieldType>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > But after updating to solr 6 shingles is not working ,schema is as
>> >> >> > below,
>> >> >> >
>> >> >> >         <fieldType name="text_shingle" class="solr.TextField"
>> >> >> > positionIncrementGap="100">
>> >> >> >         <analyzer type="index">
>> >> >> >          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> >> >> >          <filter class="solr.ShingleFilterFactory"
>> minShingleSize="2"
>> >> >> > maxShingleSize="5"
>> >> >> >                  outputUnigrams="false"
>> >> >> > outputUnigramsIfNoShingles="false" />
>> >> >> >           <filter class="solr.LowerCaseFilterFactory"/>
>> >> >> >         </analyzer>
>> >> >> >         <analyzer type="query">
>> >> >> >           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> >> >> >          <filter class="solr.ShingleFilterFactory"
>> minShingleSize="2"
>> >> >> > maxShingleSize="5"
>> >> >> >                  outputUnigrams="false"
>> >> >> > outputUnigramsIfNoShingles="false" />
>> >> >> >           <filter class="solr.LowerCaseFilterFactory"/>
>> >> >> >         </analyzer>
>> >> >> >       </fieldType>
>> >> >> >
>> >> >> > Although in the Analysis tab is was showing proper shingle result
>> but
>> >> >> > when
>> >> >> > using in the queryParser it was not giving proper results
>> >> >> >
>> >> >> > my sample hit is
>> >> >> >
>> >> >> >
>> >> >> >
>> >> <a href="http://localhost:8983/solr/shingel_test/select?q=one%">http://localhost:8983/solr/shingel_test/select?q=one%
>> 20plus%20one&wt=xml&debugQuery=true&defType=edismax&qf=cust_shingle
>> >> >> >
>> >> >> > it create the parsed query as
>> >> >> >
>> >> >> > <str name="rawquerystring">one plus one</str>
>> >> >> > <str name="querystring">one plus one</str>
>> >> >> > <str name="parsedquery">(+())/no_coord</str>
>> >> >> > <str name="parsedquery_toString">+()</str>
>> >> >> > <lst name="explain"/>
>> >> >> > <str name="QParser">ExtendedDismaxQParser</str>
>> >>
>> >>
>> >>
>>
Reply | Threaded
Open this post in threaded view
|

Re: Solr shingles is not working in solr 6.4.0

Aman Deep Singh
No it doesn't work

On 17-Mar-2017 8:38 AM, "Alexandre Rafalovitch" <[hidden email]> wrote:

> Which is what I believe you had as a working example in your Dropbox
> images.
>
> So, does it work now?
>
> Regards,
>    Alex.
> ----
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 16 March 2017 at 22:22, Aman Deep Singh <[hidden email]>
> wrote:
> > If I give query in quotes it converted query in to graph query as
> >
> > Graph(cust_sh6:"one plus one" hasBoolean=false hasPhrase=false)
> >
> >
> > On 17-Mar-2017 1:38 AM, "Alexandre Rafalovitch" <[hidden email]>
> wrote:
> >
> >> Oh. Try your query with quotes around the phone phrase:
> >> q="one plus one"
> >>
> >> My hypothesis is:
> >> Query parser splits things on whitespace before passing it down into
> >> analyzer chain as individual match attempts. The Analysis UI does not
> >> take that into account and treats the whole string as phrase sent. You
> >> say
> >> outputUnigrams="false" outputUnigramsIfNoShingles="false"
> >> So, every single token during the query gets ignored because there is
> >> nothing for it to shingle with.
> >>
> >> I am not sure why it would have worked in Solr 4.
> >>
> >> Regards,
> >>    Alex.
> >> ----
> >> http://www.solr-start.com/ - Resources for Solr users, new and
> experienced
> >>
> >>
> >> On 16 March 2017 at 13:06, Aman Deep Singh <[hidden email]>
> >> wrote:
> >> > For images dropbox url is
> >> > https://www.dropbox.com/sh/6dy6a8ajabjtxrt/
> >> AAAoxhZQe2vp3sTl3Av71_eHa?dl=0
> >> >
> >> >
> >> > On Thu, Mar 16, 2017 at 10:29 PM Aman Deep Singh <
> >> [hidden email]>
> >> > wrote:
> >> >
> >> >> Yes I have reloaded the core after config changes
> >> >>
> >> >>
> >> >> On 16-Mar-2017 10:28 PM, "Alexandre Rafalovitch" <[hidden email]
> >
> >> >> wrote:
> >> >>
> >> >> Images do not come through.
> >> >>
> >> >> But I was wrong too. You use eDismax and pass "cust_shingle" in, so
> >> >> the "df" value is irrelevant.
> >> >>
> >> >> You definitely reloaded the core after changing definitions?
> >> >> ----
> >> >> http://www.solr-start.com/ - Resources for Solr users, new and
> >> experienced
> >> >>
> >> >>
> >> >> On 16 March 2017 at 12:37, Aman Deep Singh <
> [hidden email]>
> >> >> wrote:
> >> >> > Already check that i am sending sceenshots of various senarios
> >> >> >
> >> >> >
> >> >> > On Thu, Mar 16, 2017 at 7:46 PM Alexandre Rafalovitch <
> >> >> [hidden email]>
> >> >> > wrote:
> >> >> >>
> >> >> >> Sanity check. Is your 'df' pointing at the field you think it is
> >> >> >> pointing at? It really does look like all tokens were eaten and
> >> >> >> nothing was left. But you should have seen that in the Analysis
> >> screen
> >> >> >> too, if you have the right field.
> >> >> >>
> >> >> >> Try adding echoParams=all to your request to see the full final
> >> >> >> parameter list. Maybe some parameters in initParams sections
> override
> >> >> >> your assumed config.
> >> >> >>
> >> >> >> Regards,
> >> >> >>    Alex.
> >> >> >> ----
> >> >> >> http://www.solr-start.com/ - Resources for Solr users, new and
> >> >> experienced
> >> >> >>
> >> >> >>
> >> >> >> On 16 March 2017 at 08:30, Aman Deep Singh <
> >> [hidden email]>
> >> >> >> wrote:
> >> >> >> > Hi,
> >> >> >> >
> >> >> >> > Recently I migrated from solr 4 to 6
> >> >> >> > IN solr 4 shinglefilterfactory is working correctly
> >> >> >> > my configration  i
> >> >> >> >
> >> >> >> >     <fieldType name="text_shingle" class="solr.TextField"
> >> >> >> > positionIncrementGap="100">
> >> >> >> >         <analyzer type="index">
> >> >> >> >          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> >> >> >> >          <filter class="solr.ShingleFilterFactory"
> >> minShingleSize="2"
> >> >> >> > maxShingleSize="5"
> >> >> >> >                  outputUnigrams="false"
> >> >> >> > outputUnigramsIfNoShingles="false" />
> >> >> >> >           <filter class="solr.LowerCaseFilterFactory"/>
> >> >> >> >         </analyzer>
> >> >> >> >         <analyzer type="query">
> >> >> >> >           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> >> >> >> >          <filter class="solr.ShingleFilterFactory"
> >> minShingleSize="2"
> >> >> >> > maxShingleSize="5"
> >> >> >> >                  outputUnigrams="false"
> >> >> >> > outputUnigramsIfNoShingles="false" />
> >> >> >> >           <filter class="solr.LowerCaseFilterFactory"/>
> >> >> >> >           <filter class="solr.PositionFilterFactory"/>
> >> >> >> >         </analyzer>
> >> >> >> >       </fieldType>
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > But after updating to solr 6 shingles is not working ,schema is
> as
> >> >> >> > below,
> >> >> >> >
> >> >> >> >         <fieldType name="text_shingle" class="solr.TextField"
> >> >> >> > positionIncrementGap="100">
> >> >> >> >         <analyzer type="index">
> >> >> >> >          <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> >> >> >> >          <filter class="solr.ShingleFilterFactory"
> >> minShingleSize="2"
> >> >> >> > maxShingleSize="5"
> >> >> >> >                  outputUnigrams="false"
> >> >> >> > outputUnigramsIfNoShingles="false" />
> >> >> >> >           <filter class="solr.LowerCaseFilterFactory"/>
> >> >> >> >         </analyzer>
> >> >> >> >         <analyzer type="query">
> >> >> >> >           <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> >> >> >> >          <filter class="solr.ShingleFilterFactory"
> >> minShingleSize="2"
> >> >> >> > maxShingleSize="5"
> >> >> >> >                  outputUnigrams="false"
> >> >> >> > outputUnigramsIfNoShingles="false" />
> >> >> >> >           <filter class="solr.LowerCaseFilterFactory"/>
> >> >> >> >         </analyzer>
> >> >> >> >       </fieldType>
> >> >> >> >
> >> >> >> > Although in the Analysis tab is was showing proper shingle
> result
> >> but
> >> >> >> > when
> >> >> >> > using in the queryParser it was not giving proper results
> >> >> >> >
> >> >> >> > my sample hit is
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> <a href="http://localhost:8983/solr/shingel_test/select?q=one%">http://localhost:8983/solr/shingel_test/select?q=one%
> >> 20plus%20one&wt=xml&debugQuery=true&defType=edismax&qf=cust_shingle
> >> >> >> >
> >> >> >> > it create the parsed query as
> >> >> >> >
> >> >> >> > <str name="rawquerystring">one plus one</str>
> >> >> >> > <str name="querystring">one plus one</str>
> >> >> >> > <str name="parsedquery">(+())/no_coord</str>
> >> >> >> > <str name="parsedquery_toString">+()</str>
> >> >> >> > <lst name="explain"/>
> >> >> >> > <str name="QParser">ExtendedDismaxQParser</str>
> >> >>
> >> >>
> >> >>
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr shingles is not working in solr 6.4.0

Shawn Heisey-2
In reply to this post by Alexandre Rafalovitch
On 3/16/2017 1:40 PM, Alexandre Rafalovitch wrote:
> Oh. Try your query with quotes around the phone phrase:
> q="one plus one"

That query with the fieldType the user supplied produces this, on 6.3.0
with the lucene parser:

"querystring":"test:\"one plus one\"",
"parsedquery":"MultiPhraseQuery(test:\"(one plus one plus one) plus
one\")", Looks a little odd, but maybe it's correct.
> My hypothesis is:
> Query parser splits things on whitespace before passing it down into
> analyzer chain as individual match attempts. The Analysis UI does not
> take that into account and treats the whole string as phrase sent. You
> say
> outputUnigrams="false" outputUnigramsIfNoShingles="false"
> So, every single token during the query gets ignored because there is
> nothing for it to shingle with.

Might be that.

If I change both of those unigram options to "true" then this is what I
see (also on 6.3.0, q.op is AND):

"querystring":"test:(one plus one)", "parsedquery":"+test:one +test:plus
+test:one",

The really mystifying thing is ... it works on the analysis page.  The
whitespace tokenizer should (in theory at least) produce the same tokens
on the analysis page as the query parser does before analysis, so I have
no idea why analysis and query produce different results.  During query
analysis, the whitespace tokenizer should basically be a no-op, because
the input has already been tokenized.

If I change the analysis to this (keyword instead of whitespace):

        <analyzer type="query">
          <tokenizer class="solr.KeywordTokenizerFactory"/>
          <filter class="solr.LowerCaseFilterFactory"/>
          <filter class="solr.ShingleFilterFactory" minShingleSize="2"
maxShingleSize="5"
                 outputUnigrams="false"
outputUnigramsIfNoShingles="false" />
        </analyzer>

Then the behavior is unchanged:

"querystring":"test:(one plus one)", "parsedquery":"",

> I am not sure why it would have worked in Solr 4.

I just tried it on on 4.9-SNAPSHOT, compiled 2015-05-20 from SVN
revision 1680667, and it doesn't work.  I don't remember whether this
was compiled from branch_4x or from the 4.9 branch.  Before that test, I
had tried back to 5.2.1 with the same results:

"querystring": "test:(one plus one)", "parsedquery": "", Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Solr shingles is not working in solr 6.4.0

Aman Deep Singh
I also tried in 5.2.1
for the query
http://localhost:8984/solr/test/select?q=TITLE_SH:one\%20plus\%20one&wt=xml&debugQuery=true

<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">1</int>
<lst name="params">
<str name="q">TITLE_SH:one\ plus\ one</str>
<str name="wt">xml</str>
<str name="debugQuery">true</str>
</lst>
</lst>
<result name="response" numFound="0" start="0"/>
<lst name="debug">
<str name="rawquerystring">TITLE_SH:one\ plus\ one</str>
<str name="querystring">TITLE_SH:one\ plus\ one</str>
<str name="parsedquery">
*((TITLE_SH:one plus TITLE_SH:one plus one)/no_coord) TITLE_SH:plus one*
</str>
<str name="parsedquery_toString">
(TITLE_SH:one plus TITLE_SH:one plus one) TITLE_SH:plus one
</str>
<lst name="explain"/>
<str name="QParser">LuceneQParser</str>


while in the solr 4.3.1
query
http://localhost:8983/solr/collection1/select?q=text_sh:one\%20plus\%20one&wt=xml&debugQuery=true

output is like
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">2</int>
<lst name="params">
<str name="q">text_sh:one\ plus\ one</str>
<str name="wt">xml</str>
<str name="debugQuery">true</str>
</lst>
</lst>
<result name="response" numFound="0" start="0"/>
<lst name="debug">
<str name="rawquerystring">text_sh:one\ plus\ one</str>
<str name="querystring">text_sh:one\ plus\ one</str>
<str name="parsedquery">
(text_sh:one plus text_sh:one plus one text_sh:plus one)/no_coord
</str>
<str name="parsedquery_toString">
*text_sh:one plus text_sh:one plus one text_sh:plus one*
</str>
<lst name="explain"/>
<str name="QParser">LuceneQParser</str>

On Fri, Mar 17, 2017 at 9:50 AM Shawn Heisey <[hidden email]> wrote:

> On 3/16/2017 1:40 PM, Alexandre Rafalovitch wrote:
> > Oh. Try your query with quotes around the phone phrase:
> > q="one plus one"
>
> That query with the fieldType the user supplied produces this, on 6.3.0
> with the lucene parser:
>
> "querystring":"test:\"one plus one\"",
> "parsedquery":"MultiPhraseQuery(test:\"(one plus one plus one) plus
> one\")", Looks a little odd, but maybe it's correct.
> > My hypothesis is:
> > Query parser splits things on whitespace before passing it down into
> > analyzer chain as individual match attempts. The Analysis UI does not
> > take that into account and treats the whole string as phrase sent. You
> > say
> > outputUnigrams="false" outputUnigramsIfNoShingles="false"
> > So, every single token during the query gets ignored because there is
> > nothing for it to shingle with.
>
> Might be that.
>
> If I change both of those unigram options to "true" then this is what I
> see (also on 6.3.0, q.op is AND):
>
> "querystring":"test:(one plus one)", "parsedquery":"+test:one +test:plus
> +test:one",
>
> The really mystifying thing is ... it works on the analysis page.  The
> whitespace tokenizer should (in theory at least) produce the same tokens
> on the analysis page as the query parser does before analysis, so I have
> no idea why analysis and query produce different results.  During query
> analysis, the whitespace tokenizer should basically be a no-op, because
> the input has already been tokenized.
>
> If I change the analysis to this (keyword instead of whitespace):
>
>         <analyzer type="query">
>           <tokenizer class="solr.KeywordTokenizerFactory"/>
>           <filter class="solr.LowerCaseFilterFactory"/>
>           <filter class="solr.ShingleFilterFactory" minShingleSize="2"
> maxShingleSize="5"
>                  outputUnigrams="false"
> outputUnigramsIfNoShingles="false" />
>         </analyzer>
>
> Then the behavior is unchanged:
>
> "querystring":"test:(one plus one)", "parsedquery":"",
>
> > I am not sure why it would have worked in Solr 4.
>
> I just tried it on on 4.9-SNAPSHOT, compiled 2015-05-20 from SVN
> revision 1680667, and it doesn't work.  I don't remember whether this
> was compiled from branch_4x or from the 4.9 branch.  Before that test, I
> had tried back to 5.2.1 with the same results:
>
> "querystring": "test:(one plus one)", "parsedquery": "", Thanks,
> Shawn
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr shingles is not working in solr 6.4.0

Aman Deep Singh
I found a workaround
after configuring the field type as

<fieldType name="cust_shingle" class="solr.TextField" positionIncrementGap=
"100"> <analyzer> <charFilter class="solr.PatternReplaceCharFilterFactory"
pattern=" " replacement=";"/> <tokenizer class=
"solr.PatternTokenizerFactory" pattern=";"/> <filter class=
"solr.ShingleFilterFactory" maxShingleSize="4"/> </analyzer> </fieldType>

So after giving the query as *one\ plus\ one* ,it started the creating the
shingles but for using that i have to give the query with omitting spaces
which is caused some problem in other fields ,Any way to overcome that.


On Fri, Mar 17, 2017 at 9:58 AM Aman Deep Singh <[hidden email]>
wrote:

> I also tried in 5.2.1
> for the query
>
> http://localhost:8984/solr/test/select?q=TITLE_SH:one\%20plus\%20one&wt=xml&debugQuery=true
> <http://localhost:8984/solr/test/select?q=TITLE_SH:one%5C%20plus%5C%20one&wt=xml&debugQuery=true>
>
> <response>
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">1</int>
> <lst name="params">
> <str name="q">TITLE_SH:one\ plus\ one</str>
> <str name="wt">xml</str>
> <str name="debugQuery">true</str>
> </lst>
> </lst>
> <result name="response" numFound="0" start="0"/>
> <lst name="debug">
> <str name="rawquerystring">TITLE_SH:one\ plus\ one</str>
> <str name="querystring">TITLE_SH:one\ plus\ one</str>
> <str name="parsedquery">
> *((TITLE_SH:one plus TITLE_SH:one plus one)/no_coord) TITLE_SH:plus one*
> </str>
> <str name="parsedquery_toString">
> (TITLE_SH:one plus TITLE_SH:one plus one) TITLE_SH:plus one
> </str>
> <lst name="explain"/>
> <str name="QParser">LuceneQParser</str>
>
>
> while in the solr 4.3.1
> query
>
> http://localhost:8983/solr/collection1/select?q=text_sh:one\%20plus\%20one&wt=xml&debugQuery=true
> <http://localhost:8983/solr/collection1/select?q=text_sh:one%5C%20plus%5C%20one&wt=xml&debugQuery=true>
>
> output is like
> <response>
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">2</int>
> <lst name="params">
> <str name="q">text_sh:one\ plus\ one</str>
> <str name="wt">xml</str>
> <str name="debugQuery">true</str>
> </lst>
> </lst>
> <result name="response" numFound="0" start="0"/>
> <lst name="debug">
> <str name="rawquerystring">text_sh:one\ plus\ one</str>
> <str name="querystring">text_sh:one\ plus\ one</str>
> <str name="parsedquery">
> (text_sh:one plus text_sh:one plus one text_sh:plus one)/no_coord
> </str>
> <str name="parsedquery_toString">
> *text_sh:one plus text_sh:one plus one text_sh:plus one*
> </str>
> <lst name="explain"/>
> <str name="QParser">LuceneQParser</str>
>
> On Fri, Mar 17, 2017 at 9:50 AM Shawn Heisey <[hidden email]> wrote:
>
> On 3/16/2017 1:40 PM, Alexandre Rafalovitch wrote:
> > Oh. Try your query with quotes around the phone phrase:
> > q="one plus one"
>
> That query with the fieldType the user supplied produces this, on 6.3.0
> with the lucene parser:
>
> "querystring":"test:\"one plus one\"",
> "parsedquery":"MultiPhraseQuery(test:\"(one plus one plus one) plus
> one\")", Looks a little odd, but maybe it's correct.
> > My hypothesis is:
> > Query parser splits things on whitespace before passing it down into
> > analyzer chain as individual match attempts. The Analysis UI does not
> > take that into account and treats the whole string as phrase sent. You
> > say
> > outputUnigrams="false" outputUnigramsIfNoShingles="false"
> > So, every single token during the query gets ignored because there is
> > nothing for it to shingle with.
>
> Might be that.
>
> If I change both of those unigram options to "true" then this is what I
> see (also on 6.3.0, q.op is AND):
>
> "querystring":"test:(one plus one)", "parsedquery":"+test:one +test:plus
> +test:one",
>
> The really mystifying thing is ... it works on the analysis page.  The
> whitespace tokenizer should (in theory at least) produce the same tokens
> on the analysis page as the query parser does before analysis, so I have
> no idea why analysis and query produce different results.  During query
> analysis, the whitespace tokenizer should basically be a no-op, because
> the input has already been tokenized.
>
> If I change the analysis to this (keyword instead of whitespace):
>
>         <analyzer type="query">
>           <tokenizer class="solr.KeywordTokenizerFactory"/>
>           <filter class="solr.LowerCaseFilterFactory"/>
>           <filter class="solr.ShingleFilterFactory" minShingleSize="2"
> maxShingleSize="5"
>                  outputUnigrams="false"
> outputUnigramsIfNoShingles="false" />
>         </analyzer>
>
> Then the behavior is unchanged:
>
> "querystring":"test:(one plus one)", "parsedquery":"",
>
> > I am not sure why it would have worked in Solr 4.
>
> I just tried it on on 4.9-SNAPSHOT, compiled 2015-05-20 from SVN
> revision 1680667, and it doesn't work.  I don't remember whether this
> was compiled from branch_4x or from the 4.9 branch.  Before that test, I
> had tried back to 5.2.1 with the same results:
>
> "querystring": "test:(one plus one)", "parsedquery": "", Thanks,
> Shawn
>
>