Synonym expansions w/ phrase slop exhausting memory after upgrading to SOLR 7

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Synonym expansions w/ phrase slop exhausting memory after upgrading to SOLR 7

Nick D
Hello All,

We recently upgraded from Solr 6.6 to Solr 7.7.2 and recently had spikes in
memory that eventually caused either an OOM or almost 100% utilization of
the available memory. After trying a few things, increasing the JVM heap,
making sure docValues were set for all Sort, facet fields (thought maybe
the fieldCache was blowing up), I was able to isolate a single query that
would cause the used memory to become fully exhausted and effectively
render the instance dead. After applying a timeAllowed  value to the query
and reducing the query phrase (system would crash on without throwing the
warning on longer queries containing synonyms). I was able to idenitify the
following warning in the logs:

o.a.s.s.SolrIndexSearcher Query: <____very long synonym expansion____>

the request took too long to iterate over terms. Timeout: timeoutAt:
812182664173653 (System.nanoTime(): 812182715745553),
TermsEnum=org.apache.lucene.codecs.blocktree.SegmentTermsEnum@7a0db441

I have narrowed the problem down to the following:
the way synonyms are being expaneded along with phrase slop.

With a ps=5 I get 4096 possible permutations of the phrase being searched
with because of synonyms, looking similar to:
ngs_title:"bereavement leave type build bereavement leave type data p"~5
 ngs_title:"bereavement leave type build bereavement bereavement type data
p"~5
 ngs_title:"bereavement leave type build bereavement jury duty type data
p"~5
 ngs_title:"bereavement leave type build bereavement maternity leave type
data p"~5
 ngs_title:"bereavement leave type build bereavement paternity type data
p"~5
 ngs_title:"bereavement leave type build bereavement paternity leave type
data p"~5
 ngs_title:"bereavement leave type build bereavement adoption leave type
data p"~5
 ngs_title:"bereavement leave type build jury duty maternity leave type
data p"~5
 ngs_title:"bereavement leave type build jury duty paternity type data p"~5
 ngs_title:"bereavement leave type build jury duty paternity leave type
data p"~5
 ngs_title:"bereavement leave type build jury duty adoption leave type data
p"~5
 ngs_title:"bereavement leave type build jury duty absence type data p"~5
 ngs_title:"bereavement leave type build maternity leave leave type data
p"~5
 ngs_title:"bereavement leave type build maternity leave bereavement type
data p"~5
 ngs_title:"bereavement leave type build maternity leave jury duty type
data p"~5

....

Previously in Solr 6 that same query, with the same synonyms (and query
analysis chain) would produce a parsedQuery like when using a &ps=5:
DisjunctionMaxQuery(((ngs_field_description:\"leave leave type build leave
leave type data ? p leave leave type type.enabled\"~5)^3.0 |
(ngs_title:\"leave leave type build leave leave type data ? p leave leave
type type.enabled\"~5)^10.0)

The expansion wasn't being applied to the added disjunctionMaxQuery to when
adjusting rankings with phrase slop.

In general the parsedqueries between 6 and 7 are differnet, with some new
`spanNears` showing but they don't create the memory consumpution issues
that I have seen when a large synonym expansion is happening along w/ using
a PS parameter.

I didn't see much in terms on release notes changes for synonym changes
(outside of SOW=false being the default for version . 7).

The field being opertated on has the following query analysis chain:

 <analyzer type="query">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt"/>
        <filter class="solr.SynonymGraphFilterFactory"
synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>

Not sure if there is a change in phrase slop that now takes synonyms into
account and if there is way to disable that kind of expansion or not. I am
not sure if it is related to SOLR-10980
<https://issues.apache.org/jira/plugins/servlet/mobile#issue/SOLR-10980> or
not, does seem to be related,  but referenced Solr 6 which does not do the
expansion.

Any help would be greatly appreciated.

Nick
Reply | Threaded
Open this post in threaded view
|

Re: Synonym expansions w/ phrase slop exhausting memory after upgrading to SOLR 7

Michael Gibney
This is related to this issue:
https://issues.apache.org/jira/browse/SOLR-13336

Also tangentially relevant:
https://issues.apache.org/jira/browse/LUCENE-8531
https://issues.apache.org/jira/browse/SOLR-12243

I think your options include:
1. setting slop=0, which restores SpanNearQuery as the graph phrase
query implementation (see LUCENE-8531)
2. downgrading to 7.5 would avoid the OOM, but would cause graph
phrase queries to be effectively ignored (see SOLR-12243)
3. upgrade to 8.0, which will restore the failsafe maxBooleanClauses,
avoiding OOM but returning an error code for affected queries (which
in your case sounds like most queries?) (see SOLR-13336)

Michael

On Tue, Dec 17, 2019 at 4:16 PM Nick D <[hidden email]> wrote:

>
> Hello All,
>
> We recently upgraded from Solr 6.6 to Solr 7.7.2 and recently had spikes in
> memory that eventually caused either an OOM or almost 100% utilization of
> the available memory. After trying a few things, increasing the JVM heap,
> making sure docValues were set for all Sort, facet fields (thought maybe
> the fieldCache was blowing up), I was able to isolate a single query that
> would cause the used memory to become fully exhausted and effectively
> render the instance dead. After applying a timeAllowed  value to the query
> and reducing the query phrase (system would crash on without throwing the
> warning on longer queries containing synonyms). I was able to idenitify the
> following warning in the logs:
>
> o.a.s.s.SolrIndexSearcher Query: <____very long synonym expansion____>
>
> the request took too long to iterate over terms. Timeout: timeoutAt:
> 812182664173653 (System.nanoTime(): 812182715745553),
> TermsEnum=org.apache.lucene.codecs.blocktree.SegmentTermsEnum@7a0db441
>
> I have narrowed the problem down to the following:
> the way synonyms are being expaneded along with phrase slop.
>
> With a ps=5 I get 4096 possible permutations of the phrase being searched
> with because of synonyms, looking similar to:
> ngs_title:"bereavement leave type build bereavement leave type data p"~5
>  ngs_title:"bereavement leave type build bereavement bereavement type data
> p"~5
>  ngs_title:"bereavement leave type build bereavement jury duty type data
> p"~5
>  ngs_title:"bereavement leave type build bereavement maternity leave type
> data p"~5
>  ngs_title:"bereavement leave type build bereavement paternity type data
> p"~5
>  ngs_title:"bereavement leave type build bereavement paternity leave type
> data p"~5
>  ngs_title:"bereavement leave type build bereavement adoption leave type
> data p"~5
>  ngs_title:"bereavement leave type build jury duty maternity leave type
> data p"~5
>  ngs_title:"bereavement leave type build jury duty paternity type data p"~5
>  ngs_title:"bereavement leave type build jury duty paternity leave type
> data p"~5
>  ngs_title:"bereavement leave type build jury duty adoption leave type data
> p"~5
>  ngs_title:"bereavement leave type build jury duty absence type data p"~5
>  ngs_title:"bereavement leave type build maternity leave leave type data
> p"~5
>  ngs_title:"bereavement leave type build maternity leave bereavement type
> data p"~5
>  ngs_title:"bereavement leave type build maternity leave jury duty type
> data p"~5
>
> ....
>
> Previously in Solr 6 that same query, with the same synonyms (and query
> analysis chain) would produce a parsedQuery like when using a &ps=5:
> DisjunctionMaxQuery(((ngs_field_description:\"leave leave type build leave
> leave type data ? p leave leave type type.enabled\"~5)^3.0 |
> (ngs_title:\"leave leave type build leave leave type data ? p leave leave
> type type.enabled\"~5)^10.0)
>
> The expansion wasn't being applied to the added disjunctionMaxQuery to when
> adjusting rankings with phrase slop.
>
> In general the parsedqueries between 6 and 7 are differnet, with some new
> `spanNears` showing but they don't create the memory consumpution issues
> that I have seen when a large synonym expansion is happening along w/ using
> a PS parameter.
>
> I didn't see much in terms on release notes changes for synonym changes
> (outside of SOW=false being the default for version . 7).
>
> The field being opertated on has the following query analysis chain:
>
>  <analyzer type="query">
>         <tokenizer class="solr.StandardTokenizerFactory"/>
>         <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt"/>
>         <filter class="solr.SynonymGraphFilterFactory"
> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>         <filter class="solr.LowerCaseFilterFactory"/>
>       </analyzer>
>
> Not sure if there is a change in phrase slop that now takes synonyms into
> account and if there is way to disable that kind of expansion or not. I am
> not sure if it is related to SOLR-10980
> <https://issues.apache.org/jira/plugins/servlet/mobile#issue/SOLR-10980> or
> not, does seem to be related,  but referenced Solr 6 which does not do the
> expansion.
>
> Any help would be greatly appreciated.
>
> Nick
Reply | Threaded
Open this post in threaded view
|

Re: Synonym expansions w/ phrase slop exhausting memory after upgrading to SOLR 7

Nick D
Michael,

Thank you so much, that was extremely helpful. My googlefu wasn't good
enough I guess.

1. Was my initial fix just to stop it from exploding.

2. Will be the perm solutions for now until we can get some things squared
away for 8.0.

Sounds like even in 8 there is a problem with any graph query expansion
potential still growing rather large but it just won't consume all
available memory, is that correct?

One final question, why would the maxbooleanqueries value in the solrconfig
still apply? Reading through all the jiras I thought that was supposed to
still be a fail safe, did I miss something?

Thanks again for your help,

Nick

On Wed, Dec 18, 2019, 8:10 AM Michael Gibney <[hidden email]>
wrote:

> This is related to this issue:
> https://issues.apache.org/jira/browse/SOLR-13336
>
> Also tangentially relevant:
> https://issues.apache.org/jira/browse/LUCENE-8531
> https://issues.apache.org/jira/browse/SOLR-12243
>
> I think your options include:
> 1. setting slop=0, which restores SpanNearQuery as the graph phrase
> query implementation (see LUCENE-8531)
> 2. downgrading to 7.5 would avoid the OOM, but would cause graph
> phrase queries to be effectively ignored (see SOLR-12243)
> 3. upgrade to 8.0, which will restore the failsafe maxBooleanClauses,
> avoiding OOM but returning an error code for affected queries (which
> in your case sounds like most queries?) (see SOLR-13336)
>
> Michael
>
> On Tue, Dec 17, 2019 at 4:16 PM Nick D <[hidden email]> wrote:
> >
> > Hello All,
> >
> > We recently upgraded from Solr 6.6 to Solr 7.7.2 and recently had spikes
> in
> > memory that eventually caused either an OOM or almost 100% utilization of
> > the available memory. After trying a few things, increasing the JVM heap,
> > making sure docValues were set for all Sort, facet fields (thought maybe
> > the fieldCache was blowing up), I was able to isolate a single query that
> > would cause the used memory to become fully exhausted and effectively
> > render the instance dead. After applying a timeAllowed  value to the
> query
> > and reducing the query phrase (system would crash on without throwing the
> > warning on longer queries containing synonyms). I was able to idenitify
> the
> > following warning in the logs:
> >
> > o.a.s.s.SolrIndexSearcher Query: <____very long synonym expansion____>
> >
> > the request took too long to iterate over terms. Timeout: timeoutAt:
> > 812182664173653 (System.nanoTime(): 812182715745553),
> > TermsEnum=org.apache.lucene.codecs.blocktree.SegmentTermsEnum@7a0db441
> >
> > I have narrowed the problem down to the following:
> > the way synonyms are being expaneded along with phrase slop.
> >
> > With a ps=5 I get 4096 possible permutations of the phrase being searched
> > with because of synonyms, looking similar to:
> > ngs_title:"bereavement leave type build bereavement leave type data p"~5
> >  ngs_title:"bereavement leave type build bereavement bereavement type
> data
> > p"~5
> >  ngs_title:"bereavement leave type build bereavement jury duty type data
> > p"~5
> >  ngs_title:"bereavement leave type build bereavement maternity leave type
> > data p"~5
> >  ngs_title:"bereavement leave type build bereavement paternity type data
> > p"~5
> >  ngs_title:"bereavement leave type build bereavement paternity leave type
> > data p"~5
> >  ngs_title:"bereavement leave type build bereavement adoption leave type
> > data p"~5
> >  ngs_title:"bereavement leave type build jury duty maternity leave type
> > data p"~5
> >  ngs_title:"bereavement leave type build jury duty paternity type data
> p"~5
> >  ngs_title:"bereavement leave type build jury duty paternity leave type
> > data p"~5
> >  ngs_title:"bereavement leave type build jury duty adoption leave type
> data
> > p"~5
> >  ngs_title:"bereavement leave type build jury duty absence type data p"~5
> >  ngs_title:"bereavement leave type build maternity leave leave type data
> > p"~5
> >  ngs_title:"bereavement leave type build maternity leave bereavement type
> > data p"~5
> >  ngs_title:"bereavement leave type build maternity leave jury duty type
> > data p"~5
> >
> > ....
> >
> > Previously in Solr 6 that same query, with the same synonyms (and query
> > analysis chain) would produce a parsedQuery like when using a &ps=5:
> > DisjunctionMaxQuery(((ngs_field_description:\"leave leave type build
> leave
> > leave type data ? p leave leave type type.enabled\"~5)^3.0 |
> > (ngs_title:\"leave leave type build leave leave type data ? p leave leave
> > type type.enabled\"~5)^10.0)
> >
> > The expansion wasn't being applied to the added disjunctionMaxQuery to
> when
> > adjusting rankings with phrase slop.
> >
> > In general the parsedqueries between 6 and 7 are differnet, with some new
> > `spanNears` showing but they don't create the memory consumpution issues
> > that I have seen when a large synonym expansion is happening along w/
> using
> > a PS parameter.
> >
> > I didn't see much in terms on release notes changes for synonym changes
> > (outside of SOW=false being the default for version . 7).
> >
> > The field being opertated on has the following query analysis chain:
> >
> >  <analyzer type="query">
> >         <tokenizer class="solr.StandardTokenizerFactory"/>
> >         <filter class="solr.StopFilterFactory" ignoreCase="true"
> > words="stopwords.txt"/>
> >         <filter class="solr.SynonymGraphFilterFactory"
> > synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
> >         <filter class="solr.LowerCaseFilterFactory"/>
> >       </analyzer>
> >
> > Not sure if there is a change in phrase slop that now takes synonyms into
> > account and if there is way to disable that kind of expansion or not. I
> am
> > not sure if it is related to SOLR-10980
> > <https://issues.apache.org/jira/plugins/servlet/mobile#issue/SOLR-10980>
> or
> > not, does seem to be related,  but referenced Solr 6 which does not do
> the
> > expansion.
> >
> > Any help would be greatly appreciated.
> >
> > Nick
>
Reply | Threaded
Open this post in threaded view
|

Re: Synonym expansions w/ phrase slop exhausting memory after upgrading to SOLR 7

Michael Gibney
Solution 2 (downgrade to 7.5) fixes the problem by reverting to
building proximity (SpanNear) queries that do not explode
exponentially like MultiPhraseQuery does; but note that
SpanNearQueries in 7.5 are dropped (SOLR-12243), so they have
literally no effect whatsoever (aside from the minimal cost of
building them before they get GC'd).

You are correct that there's still a problem wrt exponential graph
query expansion (a case often associated with multi-term synonyms and
WordDelimiterGraphFilter). In 8.1 and later, the parser still
*attempts* to build the exact same queries, but will be prevented from
doing so (and return an error response code) when expansion hits the
effective maxBooleanClauses threshold. So individual requests will
(for certain analysis/input combinations) still error out, but the
entire system is no longer affected. Some related issues (having to do
with graph token streams) that might be of interest:
https://issues.apache.org/jira/browse/LUCENE-8544
https://issues.apache.org/jira/browse/LUCENE-7398
https://issues.apache.org/jira/browse/LUCENE-4312


I interpret the documentation
(https://lucene.apache.org/solr/guide/8_3/query-settings-in-solrconfig.html#maxbooleanclauses)
to indicate that the maxBooleanClauses setting in solr.xml takes
priority as a hard upper bound; so the individual settings in
solrconfig.xml may be used per-collection to *decrease* the global
limit. If you attempt to set maxBooleanClauses larger in
solrconfig.xml than the systemwide setting (configured via solr.xml),
a warning message will be logged, but the attempted configuration will
otherwise have no affect (the lower systemwide value will still be the
effective limit).

Michael


On Wed, Dec 18, 2019 at 9:48 PM Nick D <[hidden email]> wrote:

>
> Michael,
>
> Thank you so much, that was extremely helpful. My googlefu wasn't good
> enough I guess.
>
> 1. Was my initial fix just to stop it from exploding.
>
> 2. Will be the perm solutions for now until we can get some things squared
> away for 8.0.
>
> Sounds like even in 8 there is a problem with any graph query expansion
> potential still growing rather large but it just won't consume all
> available memory, is that correct?
>
> One final question, why would the maxbooleanqueries value in the solrconfig
> still apply? Reading through all the jiras I thought that was supposed to
> still be a fail safe, did I miss something?
>
> Thanks again for your help,
>
> Nick
>
> On Wed, Dec 18, 2019, 8:10 AM Michael Gibney <[hidden email]>
> wrote:
>
> > This is related to this issue:
> > https://issues.apache.org/jira/browse/SOLR-13336
> >
> > Also tangentially relevant:
> > https://issues.apache.org/jira/browse/LUCENE-8531
> > https://issues.apache.org/jira/browse/SOLR-12243
> >
> > I think your options include:
> > 1. setting slop=0, which restores SpanNearQuery as the graph phrase
> > query implementation (see LUCENE-8531)
> > 2. downgrading to 7.5 would avoid the OOM, but would cause graph
> > phrase queries to be effectively ignored (see SOLR-12243)
> > 3. upgrade to 8.0, which will restore the failsafe maxBooleanClauses,
> > avoiding OOM but returning an error code for affected queries (which
> > in your case sounds like most queries?) (see SOLR-13336)
> >
> > Michael
> >
> > On Tue, Dec 17, 2019 at 4:16 PM Nick D <[hidden email]> wrote:
> > >
> > > Hello All,
> > >
> > > We recently upgraded from Solr 6.6 to Solr 7.7.2 and recently had spikes
> > in
> > > memory that eventually caused either an OOM or almost 100% utilization of
> > > the available memory. After trying a few things, increasing the JVM heap,
> > > making sure docValues were set for all Sort, facet fields (thought maybe
> > > the fieldCache was blowing up), I was able to isolate a single query that
> > > would cause the used memory to become fully exhausted and effectively
> > > render the instance dead. After applying a timeAllowed  value to the
> > query
> > > and reducing the query phrase (system would crash on without throwing the
> > > warning on longer queries containing synonyms). I was able to idenitify
> > the
> > > following warning in the logs:
> > >
> > > o.a.s.s.SolrIndexSearcher Query: <____very long synonym expansion____>
> > >
> > > the request took too long to iterate over terms. Timeout: timeoutAt:
> > > 812182664173653 (System.nanoTime(): 812182715745553),
> > > TermsEnum=org.apache.lucene.codecs.blocktree.SegmentTermsEnum@7a0db441
> > >
> > > I have narrowed the problem down to the following:
> > > the way synonyms are being expaneded along with phrase slop.
> > >
> > > With a ps=5 I get 4096 possible permutations of the phrase being searched
> > > with because of synonyms, looking similar to:
> > > ngs_title:"bereavement leave type build bereavement leave type data p"~5
> > >  ngs_title:"bereavement leave type build bereavement bereavement type
> > data
> > > p"~5
> > >  ngs_title:"bereavement leave type build bereavement jury duty type data
> > > p"~5
> > >  ngs_title:"bereavement leave type build bereavement maternity leave type
> > > data p"~5
> > >  ngs_title:"bereavement leave type build bereavement paternity type data
> > > p"~5
> > >  ngs_title:"bereavement leave type build bereavement paternity leave type
> > > data p"~5
> > >  ngs_title:"bereavement leave type build bereavement adoption leave type
> > > data p"~5
> > >  ngs_title:"bereavement leave type build jury duty maternity leave type
> > > data p"~5
> > >  ngs_title:"bereavement leave type build jury duty paternity type data
> > p"~5
> > >  ngs_title:"bereavement leave type build jury duty paternity leave type
> > > data p"~5
> > >  ngs_title:"bereavement leave type build jury duty adoption leave type
> > data
> > > p"~5
> > >  ngs_title:"bereavement leave type build jury duty absence type data p"~5
> > >  ngs_title:"bereavement leave type build maternity leave leave type data
> > > p"~5
> > >  ngs_title:"bereavement leave type build maternity leave bereavement type
> > > data p"~5
> > >  ngs_title:"bereavement leave type build maternity leave jury duty type
> > > data p"~5
> > >
> > > ....
> > >
> > > Previously in Solr 6 that same query, with the same synonyms (and query
> > > analysis chain) would produce a parsedQuery like when using a &ps=5:
> > > DisjunctionMaxQuery(((ngs_field_description:\"leave leave type build
> > leave
> > > leave type data ? p leave leave type type.enabled\"~5)^3.0 |
> > > (ngs_title:\"leave leave type build leave leave type data ? p leave leave
> > > type type.enabled\"~5)^10.0)
> > >
> > > The expansion wasn't being applied to the added disjunctionMaxQuery to
> > when
> > > adjusting rankings with phrase slop.
> > >
> > > In general the parsedqueries between 6 and 7 are differnet, with some new
> > > `spanNears` showing but they don't create the memory consumpution issues
> > > that I have seen when a large synonym expansion is happening along w/
> > using
> > > a PS parameter.
> > >
> > > I didn't see much in terms on release notes changes for synonym changes
> > > (outside of SOW=false being the default for version . 7).
> > >
> > > The field being opertated on has the following query analysis chain:
> > >
> > >  <analyzer type="query">
> > >         <tokenizer class="solr.StandardTokenizerFactory"/>
> > >         <filter class="solr.StopFilterFactory" ignoreCase="true"
> > > words="stopwords.txt"/>
> > >         <filter class="solr.SynonymGraphFilterFactory"
> > > synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
> > >         <filter class="solr.LowerCaseFilterFactory"/>
> > >       </analyzer>
> > >
> > > Not sure if there is a change in phrase slop that now takes synonyms into
> > > account and if there is way to disable that kind of expansion or not. I
> > am
> > > not sure if it is related to SOLR-10980
> > > <https://issues.apache.org/jira/plugins/servlet/mobile#issue/SOLR-10980>
> > or
> > > not, does seem to be related,  but referenced Solr 6 which does not do
> > the
> > > expansion.
> > >
> > > Any help would be greatly appreciated.
> > >
> > > Nick
> >