SynonymGraphFilter followed by StopFilter

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

SynonymGraphFilter followed by StopFilter

Andrea Gazzarini-6
Hi,
I have the following field type definition:
<fieldtype name="text" class="solr.TextField" autoGeneratePhraseQueries="true">
    <analyzer type="index">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
    <analyzer type="query">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="false" expand="true"/>
        <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="false"/>
    </analyzer>
</fieldtype>
Where synonyms and stopwords are defined as follows:

synonyms = out of warranty,oow
stopwords = of

Running the following query:

q=my tv went out of warranty something of

I get wrong results, with the following explain:

title:my title:tv title:went (title:oow PhraseQuery(title:"out ? warranty something"))

That is, the synonyms is correctly detected, I see the graph information are correctly reported in the positionLength, it seems they are wrongly interpreted by the QueryParser.
I guess the reason is the "of" removal operated by the StopFilter, which
  • removes the "of" term within the phrase (I wouldn't want that)
  • creates a "hole" in the span defined by the "oow" term, which has been marked as a synonym with a positionLength = 3, therefore including the next available term (something).
I tried to change the StopFilter in order to ignore stopwords that are marked as SYNONYM or that are part of a previous synonym span, and it works: it correctly produces the following query:

title:my title:tv title:went (title:oow PhraseQuery(title:"out of warranty")) title:something

So I'd like to ask your opinion about this. Am I missing something? Do you think it's better to open a JIRA issue? If the solution is a graph aware stop filter, do you think it's better to change the existing filter or to subclass it?

Best,
Andrea


Reply | Threaded
Open this post in threaded view
|

Re: SynonymGraphFilter followed by StopFilter

Alan Woodward-3
Hi Andrea,

This is a long-standing issue: see https://issues.apache.org/jira/browse/LUCENE-4065 and https://issues.apache.org/jira/browse/LUCENE-8250 for discussion.  I don’t think we’ve reached a consensus on how to fix it yet, but more examples are good.

Unfortunately I don’t think changing the StopFilter to ignore SYNONYM tokens will work, because then you’ll generate queries that always fail - they’ll search for ‘of’ in the middle of the phrase, but ‘of’ never gets indexed because it’s removed by the StopFilter at index time.

- Alan

On 26 Jul 2018, at 08:04, Andrea Gazzarini <[hidden email]> wrote:

Hi,
I have the following field type definition:
<fieldtype name="text" class="solr.TextField" autoGeneratePhraseQueries="true">
    <analyzer type="index">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
    <analyzer type="query">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="false" expand="true"/>
        <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="false"/>
    </analyzer>
</fieldtype>
Where synonyms and stopwords are defined as follows:

synonyms = out of warranty,oow
stopwords = of

Running the following query:

q=my tv went out of warranty something of

I get wrong results, with the following explain:

title:my title:tv title:went (title:oow PhraseQuery(title:"out ? warranty something"))

That is, the synonyms is correctly detected, I see the graph information are correctly reported in the positionLength, it seems they are wrongly interpreted by the QueryParser.
I guess the reason is the "of" removal operated by the StopFilter, which
  • removes the "of" term within the phrase (I wouldn't want that)
  • creates a "hole" in the span defined by the "oow" term, which has been marked as a synonym with a positionLength = 3, therefore including the next available term (something).
I tried to change the StopFilter in order to ignore stopwords that are marked as SYNONYM or that are part of a previous synonym span, and it works: it correctly produces the following query:

title:my title:tv title:went (title:oow PhraseQuery(title:"out of warranty")) title:something

So I'd like to ask your opinion about this. Am I missing something? Do you think it's better to open a JIRA issue? If the solution is a graph aware stop filter, do you think it's better to change the existing filter or to subclass it?

Best,
Andrea



Reply | Threaded
Open this post in threaded view
|

Re: SynonymGraphFilter followed by StopFilter

Andrea Gazzarini-6

Hi Alan, thanks for the response and thank you very much for the pointers


On 26/07/18 12:16, Alan Woodward wrote:
Hi Andrea,

This is a long-standing issue: see https://issues.apache.org/jira/browse/LUCENE-4065 and https://issues.apache.org/jira/browse/LUCENE-8250 for discussion.  I don’t think we’ve reached a consensus on how to fix it yet, but more examples are good.

Unfortunately I don’t think changing the StopFilter to ignore SYNONYM tokens will work, because then you’ll generate queries that always fail - they’ll search for ‘of’ in the middle of the phrase, but ‘of’ never gets indexed because it’s removed by the StopFilter at index time.

- Alan

On 26 Jul 2018, at 08:04, Andrea Gazzarini <[hidden email]> wrote:

Hi,
I have the following field type definition:
<fieldtype name="text" class="solr.TextField" autoGeneratePhraseQueries="true">
    <analyzer type="index">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
    <analyzer type="query">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="false" expand="true"/>
        <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="false"/>
    </analyzer>
</fieldtype>
Where synonyms and stopwords are defined as follows:

synonyms = out of warranty,oow
stopwords = of

Running the following query:

q=my tv went out of warranty something of

I get wrong results, with the following explain:

title:my title:tv title:went (title:oow PhraseQuery(title:"out ? warranty something"))

That is, the synonyms is correctly detected, I see the graph information are correctly reported in the positionLength, it seems they are wrongly interpreted by the QueryParser.
I guess the reason is the "of" removal operated by the StopFilter, which
  • removes the "of" term within the phrase (I wouldn't want that)
  • creates a "hole" in the span defined by the "oow" term, which has been marked as a synonym with a positionLength = 3, therefore including the next available term (something).
I tried to change the StopFilter in order to ignore stopwords that are marked as SYNONYM or that are part of a previous synonym span, and it works: it correctly produces the following query:

title:my title:tv title:went (title:oow PhraseQuery(title:"out of warranty")) title:something

So I'd like to ask your opinion about this. Am I missing something? Do you think it's better to open a JIRA issue? If the solution is a graph aware stop filter, do you think it's better to change the existing filter or to subclass it?

Best,
Andrea




Reply | Threaded
Open this post in threaded view
|

Re: SynonymGraphFilter followed by StopFilter

Walter Underwood
Move the synonym filter to the index analyzer chain. That provides better performance and avoids some surprising relevance behavior. With synonyms at query time, you’ll see different idf for terms in the synonym set, with the rare variant scoring higher. That is probably the opposite of what is expected.

Also, phrase synonyms just don’t work at query time because the terms are parsed into individual tokens by the query parser, not the tokenizer.

Don’t use stop words. Just remove that line. Removing stop words is a performance and space hack that was useful in the 1960’s, but causes problems now. I’ve never used stop word removal and I started in search with Infoseek in 1996. Stop word removal is like a binary idf, ignoring common words. Since we have idf, we can give a lower score to common words and keep them in the index. 

Do those two things and it should work as you expect. 

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

On Jul 26, 2018, at 3:23 AM, Andrea Gazzarini <[hidden email]> wrote:

Hi Alan, thanks for the response and thank you very much for the pointers


On 26/07/18 12:16, Alan Woodward wrote:
Hi Andrea,

This is a long-standing issue: see https://issues.apache.org/jira/browse/LUCENE-4065 and https://issues.apache.org/jira/browse/LUCENE-8250 for discussion.  I don’t think we’ve reached a consensus on how to fix it yet, but more examples are good.

Unfortunately I don’t think changing the StopFilter to ignore SYNONYM tokens will work, because then you’ll generate queries that always fail - they’ll search for ‘of’ in the middle of the phrase, but ‘of’ never gets indexed because it’s removed by the StopFilter at index time.

- Alan

On 26 Jul 2018, at 08:04, Andrea Gazzarini <[hidden email]> wrote:

Hi,
I have the following field type definition:
<fieldtype name="text" class="solr.TextField" autoGeneratePhraseQueries="true">
    <analyzer type="index">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
    <analyzer type="query">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="false" expand="true"/>
        <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="false"/>
    </analyzer>
</fieldtype>
Where synonyms and stopwords are defined as follows:

synonyms = out of warranty,oow
stopwords = of

Running the following query:

q=my tv went out of warranty something of

I get wrong results, with the following explain:

title:my title:tv title:went (title:oow PhraseQuery(title:"out ? warranty something"))

That is, the synonyms is correctly detected, I see the graph information are correctly reported in the positionLength, it seems they are wrongly interpreted by the QueryParser.
I guess the reason is the "of" removal operated by the StopFilter, which
  • removes the "of" term within the phrase (I wouldn't want that)
  • creates a "hole" in the span defined by the "oow" term, which has been marked as a synonym with a positionLength = 3, therefore including the next available term (something).
I tried to change the StopFilter in order to ignore stopwords that are marked as SYNONYM or that are part of a previous synonym span, and it works: it correctly produces the following query:

title:my title:tv title:went (title:oow PhraseQuery(title:"out of warranty")) title:something

So I'd like to ask your opinion about this. Am I missing something? Do you think it's better to open a JIRA issue? If the solution is a graph aware stop filter, do you think it's better to change the existing filter or to subclass it?

Best,
Andrea





Reply | Threaded
Open this post in threaded view
|

Re: SynonymGraphFilter followed by StopFilter

Alan Woodward-3
Also, phrase synonyms just don’t work at query time because the terms are parsed into individual tokens by the query parser, not the tokenizer.

This is no longer the case.  In general I’d avoid index-time synonyms in lucene because synonyms can create graphs (eg if a single term gets expanded to several terms), and we can’t index graphs correctly.

I’d agree that removing stop words is generally unnecessary, but there are other reasons that you’d want to filter out terms from the Tokenstream, and we should be able to handle those situations correctly.

On 26 Jul 2018, at 15:59, Walter Underwood <[hidden email]> wrote:

Move the synonym filter to the index analyzer chain. That provides better performance and avoids some surprising relevance behavior. With synonyms at query time, you’ll see different idf for terms in the synonym set, with the rare variant scoring higher. That is probably the opposite of what is expected.

Also, phrase synonyms just don’t work at query time because the terms are parsed into individual tokens by the query parser, not the tokenizer.

Don’t use stop words. Just remove that line. Removing stop words is a performance and space hack that was useful in the 1960’s, but causes problems now. I’ve never used stop word removal and I started in search with Infoseek in 1996. Stop word removal is like a binary idf, ignoring common words. Since we have idf, we can give a lower score to common words and keep them in the index. 

Do those two things and it should work as you expect. 

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

On Jul 26, 2018, at 3:23 AM, Andrea Gazzarini <[hidden email]> wrote:

Hi Alan, thanks for the response and thank you very much for the pointers


On 26/07/18 12:16, Alan Woodward wrote:
Hi Andrea,

This is a long-standing issue: see https://issues.apache.org/jira/browse/LUCENE-4065 and https://issues.apache.org/jira/browse/LUCENE-8250 for discussion.  I don’t think we’ve reached a consensus on how to fix it yet, but more examples are good.

Unfortunately I don’t think changing the StopFilter to ignore SYNONYM tokens will work, because then you’ll generate queries that always fail - they’ll search for ‘of’ in the middle of the phrase, but ‘of’ never gets indexed because it’s removed by the StopFilter at index time.

- Alan

On 26 Jul 2018, at 08:04, Andrea Gazzarini <[hidden email]> wrote:

Hi,
I have the following field type definition:
<fieldtype name="text" class="solr.TextField" autoGeneratePhraseQueries="true">
    <analyzer type="index">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
    <analyzer type="query">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="false" expand="true"/>
        <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="false"/>
    </analyzer>
</fieldtype>
Where synonyms and stopwords are defined as follows:

synonyms = out of warranty,oow
stopwords = of

Running the following query:

q=my tv went out of warranty something of

I get wrong results, with the following explain:

title:my title:tv title:went (title:oow PhraseQuery(title:"out ? warranty something"))

That is, the synonyms is correctly detected, I see the graph information are correctly reported in the positionLength, it seems they are wrongly interpreted by the QueryParser.
I guess the reason is the "of" removal operated by the StopFilter, which
  • removes the "of" term within the phrase (I wouldn't want that)
  • creates a "hole" in the span defined by the "oow" term, which has been marked as a synonym with a positionLength = 3, therefore including the next available term (something).
I tried to change the StopFilter in order to ignore stopwords that are marked as SYNONYM or that are part of a previous synonym span, and it works: it correctly produces the following query:

title:my title:tv title:went (title:oow PhraseQuery(title:"out of warranty")) title:something

So I'd like to ask your opinion about this. Am I missing something? Do you think it's better to open a JIRA issue? If the solution is a graph aware stop filter, do you think it's better to change the existing filter or to subclass it?

Best,
Andrea






Reply | Threaded
Open this post in threaded view
|

Re: SynonymGraphFilter followed by StopFilter

Andrea Gazzarini-6
In reply to this post by Walter Underwood
Hi Walter,
many thanks for the response and without any constraint at all, I would agree with you. From your message I clearly understand your experience is greater than mine. My 2 cents inline below:

> Move the synonym filter to the index analyzer chain. That provides better performance and avoids some surprising relevance behavior. With synonyms at query time, you’ll see different idf for terms in the synonym set, with the rare variant scoring higher. That is probably the opposite of what is expected.

Unfortunately moving the synonym filter to the index analyzer is not an option: the project where I'm working on has a huge index and the synonyms list is something that (at least in this stage) frequently changes; re-index everything from scratch each time a change occurs is a big problem. On the other side, the IDF issue you mention doesn't produce so many unwanted effect, at least until now. But I got the point, thanks for the hint.

> Also, phrase synonyms just don’t work at query time because the terms are parsed into individual tokens by the query parser, not the tokenizer.
Here I dont' get you: using the SynonymGraph Filter + SplitOnWhiteSpace = false + AutoGeneratePhraseQueries I get the synonym phrasing correctly working (see the first example in my email).

> Don’t use stop words. Just remove that line. Removing stop words is a performance and space hack that was useful in the 1960’s, but causes problems now. I’ve never used stop word removal and I started in search with Infoseek in 1996. Stop word removal is like a binary idf, ignoring common words. Since we have idf, we can give a lower score to common words and keep them in the index. 

And this is, as I see, something which animated long discussions around using / avoiding stopwords. I will check your suggestion, what it means to apply that approach to my project, but in meantime I think, also looking at the JIRA Alan pointed in his answer, the issue is there, and it's real; I mean, it is something that it doesn't work as expected (my use case, as far as I understand, is just an example because the thing is broader and it is related to the FilteredTokenFilter)

Thanks again,
Andrea

On 26/07/18 16:59, Walter Underwood wrote:
Move the synonym filter to the index analyzer chain. That provides better performance and avoids some surprising relevance behavior. With synonyms at query time, you’ll see different idf for terms in the synonym set, with the rare variant scoring higher. That is probably the opposite of what is expected.

Also, phrase synonyms just don’t work at query time because the terms are parsed into individual tokens by the query parser, not the tokenizer.

Don’t use stop words. Just remove that line. Removing stop words is a performance and space hack that was useful in the 1960’s, but causes problems now. I’ve never used stop word removal and I started in search with Infoseek in 1996. Stop word removal is like a binary idf, ignoring common words. Since we have idf, we can give a lower score to common words and keep them in the index. 

Do those two things and it should work as you expect. 

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

On Jul 26, 2018, at 3:23 AM, Andrea Gazzarini <[hidden email]> wrote:

Hi Alan, thanks for the response and thank you very much for the pointers


On 26/07/18 12:16, Alan Woodward wrote:
Hi Andrea,

This is a long-standing issue: see https://issues.apache.org/jira/browse/LUCENE-4065 and https://issues.apache.org/jira/browse/LUCENE-8250 for discussion.  I don’t think we’ve reached a consensus on how to fix it yet, but more examples are good.

Unfortunately I don’t think changing the StopFilter to ignore SYNONYM tokens will work, because then you’ll generate queries that always fail - they’ll search for ‘of’ in the middle of the phrase, but ‘of’ never gets indexed because it’s removed by the StopFilter at index time.

- Alan

On 26 Jul 2018, at 08:04, Andrea Gazzarini <[hidden email]> wrote:

Hi,
I have the following field type definition:
<fieldtype name="text" class="solr.TextField" autoGeneratePhraseQueries="true">
    <analyzer type="index">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
    <analyzer type="query">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="false" expand="true"/>
        <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="false"/>
    </analyzer>
</fieldtype>
Where synonyms and stopwords are defined as follows:

synonyms = out of warranty,oow
stopwords = of

Running the following query:

q=my tv went out of warranty something of

I get wrong results, with the following explain:

title:my title:tv title:went (title:oow PhraseQuery(title:"out ? warranty something"))

That is, the synonyms is correctly detected, I see the graph information are correctly reported in the positionLength, it seems they are wrongly interpreted by the QueryParser.
I guess the reason is the "of" removal operated by the StopFilter, which
  • removes the "of" term within the phrase (I wouldn't want that)
  • creates a "hole" in the span defined by the "oow" term, which has been marked as a synonym with a positionLength = 3, therefore including the next available term (something).
I tried to change the StopFilter in order to ignore stopwords that are marked as SYNONYM or that are part of a previous synonym span, and it works: it correctly produces the following query:

title:my title:tv title:went (title:oow PhraseQuery(title:"out of warranty")) title:something

So I'd like to ask your opinion about this. Am I missing something? Do you think it's better to open a JIRA issue? If the solution is a graph aware stop filter, do you think it's better to change the existing filter or to subclass it?

Best,
Andrea






Reply | Threaded
Open this post in threaded view
|

Re: SynonymGraphFilter followed by StopFilter

Michael Sokolov-4
 > In general I’d avoid index-time synonyms in lucene because synonyms can create graphs (eg if a single term gets expanded to several terms), and we can’t index graphs correctly.

I wonder what it would take to address this. I guess the blast radius of adding a token "width" could be pretty large. Is there an issue or any past discussion about that?

On Thu, Jul 26, 2018 at 11:42 AM Andrea Gazzarini <[hidden email]> wrote:
Hi Walter,
many thanks for the response and without any constraint at all, I would agree with you. From your message I clearly understand your experience is greater than mine. My 2 cents inline below:

> Move the synonym filter to the index analyzer chain. That provides better performance and avoids some surprising relevance behavior. With synonyms at query time, you’ll see different idf for terms in the synonym set, with the rare variant scoring higher. That is probably the opposite of what is expected.

Unfortunately moving the synonym filter to the index analyzer is not an option: the project where I'm working on has a huge index and the synonyms list is something that (at least in this stage) frequently changes; re-index everything from scratch each time a change occurs is a big problem. On the other side, the IDF issue you mention doesn't produce so many unwanted effect, at least until now. But I got the point, thanks for the hint.

> Also, phrase synonyms just don’t work at query time because the terms are parsed into individual tokens by the query parser, not the tokenizer.
Here I dont' get you: using the SynonymGraph Filter + SplitOnWhiteSpace = false + AutoGeneratePhraseQueries I get the synonym phrasing correctly working (see the first example in my email).

> Don’t use stop words. Just remove that line. Removing stop words is a performance and space hack that was useful in the 1960’s, but causes problems now. I’ve never used stop word removal and I started in search with Infoseek in 1996. Stop word removal is like a binary idf, ignoring common words. Since we have idf, we can give a lower score to common words and keep them in the index. 

And this is, as I see, something which animated long discussions around using / avoiding stopwords. I will check your suggestion, what it means to apply that approach to my project, but in meantime I think, also looking at the JIRA Alan pointed in his answer, the issue is there, and it's real; I mean, it is something that it doesn't work as expected (my use case, as far as I understand, is just an example because the thing is broader and it is related to the FilteredTokenFilter)

Thanks again,
Andrea

On 26/07/18 16:59, Walter Underwood wrote:
Move the synonym filter to the index analyzer chain. That provides better performance and avoids some surprising relevance behavior. With synonyms at query time, you’ll see different idf for terms in the synonym set, with the rare variant scoring higher. That is probably the opposite of what is expected.

Also, phrase synonyms just don’t work at query time because the terms are parsed into individual tokens by the query parser, not the tokenizer.

Don’t use stop words. Just remove that line. Removing stop words is a performance and space hack that was useful in the 1960’s, but causes problems now. I’ve never used stop word removal and I started in search with Infoseek in 1996. Stop word removal is like a binary idf, ignoring common words. Since we have idf, we can give a lower score to common words and keep them in the index. 

Do those two things and it should work as you expect. 

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

On Jul 26, 2018, at 3:23 AM, Andrea Gazzarini <[hidden email]> wrote:

Hi Alan, thanks for the response and thank you very much for the pointers


On 26/07/18 12:16, Alan Woodward wrote:
Hi Andrea,

This is a long-standing issue: see https://issues.apache.org/jira/browse/LUCENE-4065 and https://issues.apache.org/jira/browse/LUCENE-8250 for discussion.  I don’t think we’ve reached a consensus on how to fix it yet, but more examples are good.

Unfortunately I don’t think changing the StopFilter to ignore SYNONYM tokens will work, because then you’ll generate queries that always fail - they’ll search for ‘of’ in the middle of the phrase, but ‘of’ never gets indexed because it’s removed by the StopFilter at index time.

- Alan

On 26 Jul 2018, at 08:04, Andrea Gazzarini <[hidden email]> wrote:

Hi,
I have the following field type definition:
<fieldtype name="text" class="solr.TextField" autoGeneratePhraseQueries="true">
    <analyzer type="index">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
    <analyzer type="query">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="false" expand="true"/>
        <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="false"/>
    </analyzer>
</fieldtype>
Where synonyms and stopwords are defined as follows:

synonyms = out of warranty,oow
stopwords = of

Running the following query:

q=my tv went out of warranty something of

I get wrong results, with the following explain:

title:my title:tv title:went (title:oow PhraseQuery(title:"out ? warranty something"))

That is, the synonyms is correctly detected, I see the graph information are correctly reported in the positionLength, it seems they are wrongly interpreted by the QueryParser.
I guess the reason is the "of" removal operated by the StopFilter, which
  • removes the "of" term within the phrase (I wouldn't want that)
  • creates a "hole" in the span defined by the "oow" term, which has been marked as a synonym with a positionLength = 3, therefore including the next available term (something).
I tried to change the StopFilter in order to ignore stopwords that are marked as SYNONYM or that are part of a previous synonym span, and it works: it correctly produces the following query:

title:my title:tv title:went (title:oow PhraseQuery(title:"out of warranty")) title:something

So I'd like to ask your opinion about this. Am I missing something? Do you think it's better to open a JIRA issue? If the solution is a graph aware stop filter, do you think it's better to change the existing filter or to subclass it?

Best,
Andrea






Reply | Threaded
Open this post in threaded view
|

Re: SynonymGraphFilter followed by StopFilter

Alessandro Benedetti
Hi all,
I just want to add that 
"With synonyms at query time, you’ll see different idf for terms in the synonym set, with the rare variant scoring higher. That is probably the opposite of what is expected."

At least that feature brings flexibility in.

Cheers

--------------------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director

On Fri, Jul 27, 2018 at 3:25 AM, Michael Sokolov <[hidden email]> wrote:
 > In general I’d avoid index-time synonyms in lucene because synonyms can create graphs (eg if a single term gets expanded to several terms), and we can’t index graphs correctly.

I wonder what it would take to address this. I guess the blast radius of adding a token "width" could be pretty large. Is there an issue or any past discussion about that?

On Thu, Jul 26, 2018 at 11:42 AM Andrea Gazzarini <[hidden email]> wrote:
Hi Walter,
many thanks for the response and without any constraint at all, I would agree with you. From your message I clearly understand your experience is greater than mine. My 2 cents inline below:

> Move the synonym filter to the index analyzer chain. That provides better performance and avoids some surprising relevance behavior. With synonyms at query time, you’ll see different idf for terms in the synonym set, with the rare variant scoring higher. That is probably the opposite of what is expected.

Unfortunately moving the synonym filter to the index analyzer is not an option: the project where I'm working on has a huge index and the synonyms list is something that (at least in this stage) frequently changes; re-index everything from scratch each time a change occurs is a big problem. On the other side, the IDF issue you mention doesn't produce so many unwanted effect, at least until now. But I got the point, thanks for the hint.

> Also, phrase synonyms just don’t work at query time because the terms are parsed into individual tokens by the query parser, not the tokenizer.
Here I dont' get you: using the SynonymGraph Filter + SplitOnWhiteSpace = false + AutoGeneratePhraseQueries I get the synonym phrasing correctly working (see the first example in my email).

> Don’t use stop words. Just remove that line. Removing stop words is a performance and space hack that was useful in the 1960’s, but causes problems now. I’ve never used stop word removal and I started in search with Infoseek in 1996. Stop word removal is like a binary idf, ignoring common words. Since we have idf, we can give a lower score to common words and keep them in the index. 

And this is, as I see, something which animated long discussions around using / avoiding stopwords. I will check your suggestion, what it means to apply that approach to my project, but in meantime I think, also looking at the JIRA Alan pointed in his answer, the issue is there, and it's real; I mean, it is something that it doesn't work as expected (my use case, as far as I understand, is just an example because the thing is broader and it is related to the FilteredTokenFilter)

Thanks again,
Andrea

On 26/07/18 16:59, Walter Underwood wrote:
Move the synonym filter to the index analyzer chain. That provides better performance and avoids some surprising relevance behavior. With synonyms at query time, you’ll see different idf for terms in the synonym set, with the rare variant scoring higher. That is probably the opposite of what is expected.

Also, phrase synonyms just don’t work at query time because the terms are parsed into individual tokens by the query parser, not the tokenizer.

Don’t use stop words. Just remove that line. Removing stop words is a performance and space hack that was useful in the 1960’s, but causes problems now. I’ve never used stop word removal and I started in search with Infoseek in 1996. Stop word removal is like a binary idf, ignoring common words. Since we have idf, we can give a lower score to common words and keep them in the index. 

Do those two things and it should work as you expect. 

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

On Jul 26, 2018, at 3:23 AM, Andrea Gazzarini <[hidden email]> wrote:

Hi Alan, thanks for the response and thank you very much for the pointers


On 26/07/18 12:16, Alan Woodward wrote:
Hi Andrea,

This is a long-standing issue: see https://issues.apache.org/jira/browse/LUCENE-4065 and https://issues.apache.org/jira/browse/LUCENE-8250 for discussion.  I don’t think we’ve reached a consensus on how to fix it yet, but more examples are good.

Unfortunately I don’t think changing the StopFilter to ignore SYNONYM tokens will work, because then you’ll generate queries that always fail - they’ll search for ‘of’ in the middle of the phrase, but ‘of’ never gets indexed because it’s removed by the StopFilter at index time.

- Alan

On 26 Jul 2018, at 08:04, Andrea Gazzarini <[hidden email]> wrote:

Hi,
I have the following field type definition:
<fieldtype name="text" class="solr.TextField" autoGeneratePhraseQueries="true">
    <analyzer type="index">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
    <analyzer type="query">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="false" expand="true"/>
        <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="false"/>
    </analyzer>
</fieldtype>
Where synonyms and stopwords are defined as follows:

synonyms = out of warranty,oow
stopwords = of

Running the following query:

q=my tv went out of warranty something of

I get wrong results, with the following explain:

title:my title:tv title:went (title:oow PhraseQuery(title:"out ? warranty something"))

That is, the synonyms is correctly detected, I see the graph information are correctly reported in the positionLength, it seems they are wrongly interpreted by the QueryParser.
I guess the reason is the "of" removal operated by the StopFilter, which
  • removes the "of" term within the phrase (I wouldn't want that)
  • creates a "hole" in the span defined by the "oow" term, which has been marked as a synonym with a positionLength = 3, therefore including the next available term (something).
I tried to change the StopFilter in order to ignore stopwords that are marked as SYNONYM or that are part of a previous synonym span, and it works: it correctly produces the following query:

title:my title:tv title:went (title:oow PhraseQuery(title:"out of warranty")) title:something

So I'd like to ask your opinion about this. Am I missing something? Do you think it's better to open a JIRA issue? If the solution is a graph aware stop filter, do you think it's better to change the existing filter or to subclass it?

Best,
Andrea







---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
Reply | Threaded
Open this post in threaded view
|

Re: SynonymGraphFilter followed by StopFilter

Robert Muir
No Solr patches necessary: synonymquery fixed that IDF issue 3 years ago. There is just extremely outdated advice on this thread.

On Fri, Jul 27, 2018 at 7:08 AM, Alessandro Benedetti <[hidden email]> wrote:
Hi all,
I just want to add that 
"With synonyms at query time, you’ll see different idf for terms in the synonym set, with the rare variant scoring higher. That is probably the opposite of what is expected."

At least that feature brings flexibility in.

Cheers

--------------------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director

On Fri, Jul 27, 2018 at 3:25 AM, Michael Sokolov <[hidden email]> wrote:
 > In general I’d avoid index-time synonyms in lucene because synonyms can create graphs (eg if a single term gets expanded to several terms), and we can’t index graphs correctly.

I wonder what it would take to address this. I guess the blast radius of adding a token "width" could be pretty large. Is there an issue or any past discussion about that?

On Thu, Jul 26, 2018 at 11:42 AM Andrea Gazzarini <[hidden email]> wrote:
Hi Walter,
many thanks for the response and without any constraint at all, I would agree with you. From your message I clearly understand your experience is greater than mine. My 2 cents inline below:

> Move the synonym filter to the index analyzer chain. That provides better performance and avoids some surprising relevance behavior. With synonyms at query time, you’ll see different idf for terms in the synonym set, with the rare variant scoring higher. That is probably the opposite of what is expected.

Unfortunately moving the synonym filter to the index analyzer is not an option: the project where I'm working on has a huge index and the synonyms list is something that (at least in this stage) frequently changes; re-index everything from scratch each time a change occurs is a big problem. On the other side, the IDF issue you mention doesn't produce so many unwanted effect, at least until now. But I got the point, thanks for the hint.

> Also, phrase synonyms just don’t work at query time because the terms are parsed into individual tokens by the query parser, not the tokenizer.
Here I dont' get you: using the SynonymGraph Filter + SplitOnWhiteSpace = false + AutoGeneratePhraseQueries I get the synonym phrasing correctly working (see the first example in my email).

> Don’t use stop words. Just remove that line. Removing stop words is a performance and space hack that was useful in the 1960’s, but causes problems now. I’ve never used stop word removal and I started in search with Infoseek in 1996. Stop word removal is like a binary idf, ignoring common words. Since we have idf, we can give a lower score to common words and keep them in the index. 

And this is, as I see, something which animated long discussions around using / avoiding stopwords. I will check your suggestion, what it means to apply that approach to my project, but in meantime I think, also looking at the JIRA Alan pointed in his answer, the issue is there, and it's real; I mean, it is something that it doesn't work as expected (my use case, as far as I understand, is just an example because the thing is broader and it is related to the FilteredTokenFilter)

Thanks again,
Andrea

On 26/07/18 16:59, Walter Underwood wrote:
Move the synonym filter to the index analyzer chain. That provides better performance and avoids some surprising relevance behavior. With synonyms at query time, you’ll see different idf for terms in the synonym set, with the rare variant scoring higher. That is probably the opposite of what is expected.

Also, phrase synonyms just don’t work at query time because the terms are parsed into individual tokens by the query parser, not the tokenizer.

Don’t use stop words. Just remove that line. Removing stop words is a performance and space hack that was useful in the 1960’s, but causes problems now. I’ve never used stop word removal and I started in search with Infoseek in 1996. Stop word removal is like a binary idf, ignoring common words. Since we have idf, we can give a lower score to common words and keep them in the index. 

Do those two things and it should work as you expect. 

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

On Jul 26, 2018, at 3:23 AM, Andrea Gazzarini <[hidden email]> wrote:

Hi Alan, thanks for the response and thank you very much for the pointers


On 26/07/18 12:16, Alan Woodward wrote:
Hi Andrea,

This is a long-standing issue: see https://issues.apache.org/jira/browse/LUCENE-4065 and https://issues.apache.org/jira/browse/LUCENE-8250 for discussion.  I don’t think we’ve reached a consensus on how to fix it yet, but more examples are good.

Unfortunately I don’t think changing the StopFilter to ignore SYNONYM tokens will work, because then you’ll generate queries that always fail - they’ll search for ‘of’ in the middle of the phrase, but ‘of’ never gets indexed because it’s removed by the StopFilter at index time.

- Alan

On 26 Jul 2018, at 08:04, Andrea Gazzarini <[hidden email]> wrote:

Hi,
I have the following field type definition:
<fieldtype name="text" class="solr.TextField" autoGeneratePhraseQueries="true">
    <analyzer type="index">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
    <analyzer type="query">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="false" expand="true"/>
        <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="false"/>
    </analyzer>
</fieldtype>
Where synonyms and stopwords are defined as follows:

synonyms = out of warranty,oow
stopwords = of

Running the following query:

q=my tv went out of warranty something of

I get wrong results, with the following explain:

title:my title:tv title:went (title:oow PhraseQuery(title:"out ? warranty something"))

That is, the synonyms is correctly detected, I see the graph information are correctly reported in the positionLength, it seems they are wrongly interpreted by the QueryParser.
I guess the reason is the "of" removal operated by the StopFilter, which
  • removes the "of" term within the phrase (I wouldn't want that)
  • creates a "hole" in the span defined by the "oow" term, which has been marked as a synonym with a positionLength = 3, therefore including the next available term (something).
I tried to change the StopFilter in order to ignore stopwords that are marked as SYNONYM or that are part of a previous synonym span, and it works: it correctly produces the following query:

title:my title:tv title:went (title:oow PhraseQuery(title:"out of warranty")) title:something

So I'd like to ask your opinion about this. Am I missing something? Do you think it's better to open a JIRA issue? If the solution is a graph aware stop filter, do you think it's better to change the existing filter or to subclass it?

Best,
Andrea








Reply | Threaded
Open this post in threaded view
|

Re: SynonymGraphFilter followed by StopFilter

Michael McCandless-2
In reply to this post by Michael Sokolov-4
On Thu, Jul 26, 2018 at 10:25 PM, Michael Sokolov <[hidden email]> wrote:
 > In general I’d avoid index-time synonyms in lucene because synonyms can create graphs (eg if a single term gets expanded to several terms), and we can’t index graphs correctly.

I wonder what it would take to address this. I guess the blast radius of adding a token "width" could be pretty large. Is there an issue or any past discussion about that?

I think some ideas have been mentioned on past issues, e.g. using payloads to hold the position length should be workable (with a custom Query) without any source code changes to Lucene, but I don't know of anyone building a prototype.