[jira] [Commented] (LUCENE-7698) CommonGramsQueryFilter in the query analyzer chain breaks phrase queries

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[jira] [Commented] (LUCENE-7698) CommonGramsQueryFilter in the query analyzer chain breaks phrase queries

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-7698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15871777#comment-15871777 ]

Michael McCandless commented on LUCENE-7698:

OK I see what's happening: this filter ({{CommonGramsQueryFilter}}) deletes the unigram tokens, but keeps {{posLength=2}} on the bigram tokens, which makes a disconnected graph, and then the query parser does the wrong thing.

I think the right fix is for it to set {{posLength}} to 1 when it drops unigram tokens .. I'll work on a patch.

> CommonGramsQueryFilter in the query analyzer chain breaks phrase queries
> ------------------------------------------------------------------------
>                 Key: LUCENE-7698
>                 URL: https://issues.apache.org/jira/browse/LUCENE-7698
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/queryparser
>    Affects Versions: 6.4, 6.4.1
>            Reporter: Ere Maijala
>              Labels: regression
> (Please pardon me if the project or component are wrong!)
> CommonGramsQueryFilter breaks phrase queries. The behavior also seems to change with addition or removal of adjacent terms.
> Steps to reproduce:
> 1.) Download and extract Solr (in my test case version 6.4.1) somewhere.
> 2.) Modify server/solr/configsets/sample_techproducts_configs/conf/managed-schema and modify text_general fieldType by adding CommonGrams(Query)Filter before stopWordFilter:
>     <fieldType name="text_general" class="solr.TextField" positionIncrementGap="100">
>       <analyzer type="index">
>         <tokenizer class="solr.StandardTokenizerFactory"/>
>         <filter class="solr.CommonGramsFilterFactory" ignoreCase="true" words="stopwords.txt" />
>         <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
>         <!-- in this example, we will only use synonyms at query time
>         <filter class="solr.SynonymFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
>         -->
>         <filter class="solr.LowerCaseFilterFactory"/>
>       </analyzer>
>       <analyzer type="query">
>         <tokenizer class="solr.StandardTokenizerFactory"/>
>         <filter class="solr.CommonGramsQueryFilterFactory" ignoreCase="true" words="stopwords.txt"/>
>         <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
>         <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>         <filter class="solr.LowerCaseFilterFactory"/>
>       </analyzer>
>     </fieldType>
> 3.) Add "with" to server/solr/configsets/sample_techproducts_configs/conf/stopwords.txt and make sure the file has correct line endings (extracted from Solr zip it seems to contain DOS/Windows lien endings which may break things).
> 4.) Run the techproducts example with "bin/solr -e techproducts"
> 5.) Browse to <http://localhost:8983/solr/techproducts/select?q=%22iPod%20with%20Video%22&debugQuery=true>
> 6.) Observe that parsedquery in the debug output is empty
> 7.) Browse to <http://localhost:8983/solr/techproducts/select?q=%22Apple%2060%20GB%20iPod%20with%20Video%20Playback%20Black%22&debugQuery=true>
> 8.) Observe that parsedquery contains ipod_with as expected but not with_video.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]