Problems using a suggester component in the /select handler in cloud mode

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Problems using a suggester component in the /select handler in cloud mode

Alexandros Paramythis
Hi everyone,

Environment:

Solr 7.5.0, cloud mode (but problem should be identical in multiple
versions, at least in 7.x)

Summary:

We have a Solr configuration that returns suggestions in the course of a
normal search call (i.e., we have a 'suggest' component added to the
'last-components' for '/select' request handler). This does not work in
cloud mode, where we get an NPE in QueryComponent. This problem seems to
have been reported in various forms in the past -- see for example [1]
and [2] (links at the end of this email) -- but we couldn't find any
resolution (or in-depth discussion for that matter).

In more detail:

We have a suggest component configured as follows:

   <searchComponent name="suggest" class="solr.SuggestComponent" >

         <lst name="suggester">
                 <str name="name">default</str>
                 <str
name="classname">org.apache.solr.spelling.suggest.Suggester</str>
                 <str
name="lookupImpl">org.apache.solr.spelling.suggest.fst.AnalyzingLookupFactory</str>
                 <str name="storeDir">dict_default</str>
                 <str name="suggestAnalyzerFieldType">text_suggest</str>
                 <str name="field">text_suggest</str>
                 <str name="buildOnStartup">true</str>
                 <str name="buildOnCommit">true</str>
                 <str name="buildOnOptimize">true</str>
         </lst>

         <lst name="suggester">
                 <str name="name">suggest_phrase</str>
                 <str
name="lookupImpl">org.apache.solr.spelling.suggest.fst.AnalyzingLookupFactory</str>
                 <str name="storeDir">dict_suggest_phrase</str>
                 <str
name="suggestAnalyzerFieldType">text_suggest_phrase</str>
                 <str name="field">suggest_phrase</str>
                 <str name="buildOnStartup">true</str>
                 <str name="buildOnCommit">true</str>
                 <str name="buildOnOptimize">true</str>
         </lst>

         <lst name="suggester">
                 <str name="name">suggest_infix_shingle</str>
                 <str name="lookupImpl">AnalyzingInfixLookupFactory</str>
                 <str name="indexPath">suggestInfixShingleDir</str>
                 <str
name="suggestAnalyzerFieldType">text_suggest_phrase</str>
                 <str name="field">suggest_phrase</str>
                 <str name="buildOnOptimize">true</str>
                 <str name="buildOnCommit">true</str>
                 <str name="buildOnStartup">true</str>
         </lst>

         <lst name="suggester">
                 <str name="name">suggest_prefix</str>
                 <str name="classname">Suggester</str>
                 <str name="lookupImpl">AnalyzingLookupFactory</str>
                 <str
name="suggestAnalyzerFieldType">text_suggest_prefix</str>
                 <str name="field">suggest_prefix</str>
                 <str name="buildOnOptimize">true</str>
                 <str name="buildOnCommit">true</str>
                 <str name="buildOnStartup">true</str>
         </lst>

   </searchComponent>


This component works without issued both in standalone and cloud mode,
when used as the sole component in a handler, such as in the following
excerpt:

         <requestHandler name="/suggest" class="solr.SearchHandler"
startup="lazy">
                 <lst name="defaults">
                     <str name="suggest.dictionary">default</str>
                     <str name="suggest.dictionary">suggest_phrase</str>
                     <str
name="suggest.dictionary">suggest_infix_shingle</str>
                     <str name="suggest.dictionary">suggest_prefix</str>
                     <str name="suggest">true</str>
                     <str name="suggest.count">10</str>
                     <str name="suggest.highlight">false</str>
                 </lst>
                 <arr name="components">
                         <str>suggest</str>
                 </arr>
         </requestHandler>


It also works when used along with other component in standalone mode,
such as in the following excerpt, where we use the suggest component to
get suggestions during a "normal" search call:

         <requestHandler name="/select" class="solr.SearchHandler">
                 <lst name="defaults">
                         <str name="echoParams">explicit</str>
                         <int name="rows">10</int>
                         <str name="df">text_search</str>

                         <str name="defType">edismax</str>

                         <str name="qf">title^5.0 subtitle^3.0
abstract^2.0 text_search</str>
                         <str name="pf">title^5.0 subtitle^3.0
abstract^2.0 text_search</str>
                         <str name="ps">4</str>
                         <str name="spellcheck">on</str>
                         <str name="spellcheck.dictionary">default</str>
                         <str name="spellcheck.extendedResults">true</str>
                         <str name="spellcheck.count">10</str>
                         <str name="spellcheck.alternativeTermCount">5</str>
                         <str name="spellcheck.maxResultsForSuggest">5</str>
                         <str name="spellcheck.collate">true</str>
                         <str
name="spellcheck.collateExtendedResults">true</str>
                         <str name="spellcheck.maxCollationTries">10</str>
                         <str name="spellcheck.maxCollations">5</str>

                         <str name="suggest.dictionary">default</str>
                         <str name="suggest.dictionary">suggest_phrase</str>
                         <str
name="suggest.dictionary">suggest_infix_shingle</str>
                         <str name="suggest.dictionary">suggest_prefix</str>
                         <str name="suggest">true</str>
                         <str name="suggest.count">10</str>
                         <str name="suggest.highlight">false</str>
                 </lst>

                 <arr name="last-components">
                         <str>suggest</str>
                         <str>spellcheck</str>
                 </arr>
         </requestHandler>

However, the above configuration does not work in cloud mode, where we
get an NPE if a search call is made:

  o.a.s.s.HttpSolrCall null:java.lang.NullPointerException
        at org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1034)
        at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:885)
        at org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:585)
        at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:564)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:426)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)


As far as we can tell, this is due to the fact that the failing code in
QueryComponent assumes that the shard response always contains a
'sort_values' section:

         NamedList sortFieldValues =
(NamedList)(srsp.getSolrResponse().getResponse().get("sort_values"));
         NamedList unmarshalledSortFieldValues = unmarshalSortValues(ss,
sortFieldValues, schema);

(the exception occurs in unmarshalSortValues(), which assumes the
sortFieldValues parameter is not null)

Are we doing something wrong that is preventing 'sort_values' from being
present in the shard response? Or is their absence OK, but the code
fails to account for it? Our guess is the second, because
umarshalSortValues() does check if the values are empty or not, and
handles that.

Can anyone help us with this issue? Thanks in advance for any pointers!

Alex

[1]
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201711.mbox/%3CCACKWnqOv=HFQwTxd7SwEqSTj-im+43FE+c9_zbTCPMWsp8vL_w@...%3E

[2] https://issues.apache.org/jira/browse/SOLR-12060

Reply | Threaded
Open this post in threaded view
|

Re: Problems using a suggester component in the /select handler in cloud mode

Alexandros Paramythis
Hi everyone,

We have a fix for the problem described in the message below. Could anyone provide pointers to documentation on how we would go about contributing this back?

Thanks in advance for your input,

Alex


On 26/06/2019 10:48, Alexandros Paramythis wrote:
Hi everyone,

Environment:

Solr 7.5.0, cloud mode (but problem should be identical in multiple versions, at least in 7.x)

Summary:

We have a Solr configuration that returns suggestions in the course of a normal search call (i.e., we have a 'suggest' component added to the 'last-components' for '/select' request handler). This does not work in cloud mode, where we get an NPE in QueryComponent. This problem seems to have been reported in various forms in the past -- see for example [1] and [2] (links at the end of this email) -- but we couldn't find any resolution (or in-depth discussion for that matter).

In more detail:

We have a suggest component configured as follows:

  <searchComponent name="suggest" class="solr.SuggestComponent" >

        <lst name="suggester">
                <str name="name">default</str>
                <str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
                <str name="lookupImpl">org.apache.solr.spelling.suggest.fst.AnalyzingLookupFactory</str>
                <str name="storeDir">dict_default</str>
                <str name="suggestAnalyzerFieldType">text_suggest</str>
                <str name="field">text_suggest</str>
                <str name="buildOnStartup">true</str>
                <str name="buildOnCommit">true</str>
                <str name="buildOnOptimize">true</str>
        </lst>

        <lst name="suggester">
                <str name="name">suggest_phrase</str>
                <str name="lookupImpl">org.apache.solr.spelling.suggest.fst.AnalyzingLookupFactory</str>
                <str name="storeDir">dict_suggest_phrase</str>
                <str name="suggestAnalyzerFieldType">text_suggest_phrase</str>
                <str name="field">suggest_phrase</str>
                <str name="buildOnStartup">true</str>
                <str name="buildOnCommit">true</str>
                <str name="buildOnOptimize">true</str>
        </lst>

        <lst name="suggester">
                <str name="name">suggest_infix_shingle</str>
                <str name="lookupImpl">AnalyzingInfixLookupFactory</str>
                <str name="indexPath">suggestInfixShingleDir</str>
                <str name="suggestAnalyzerFieldType">text_suggest_phrase</str>
                <str name="field">suggest_phrase</str>
                <str name="buildOnOptimize">true</str>
                <str name="buildOnCommit">true</str>
                <str name="buildOnStartup">true</str>
        </lst>

        <lst name="suggester">
                <str name="name">suggest_prefix</str>
                <str name="classname">Suggester</str>
                <str name="lookupImpl">AnalyzingLookupFactory</str>
                <str name="suggestAnalyzerFieldType">text_suggest_prefix</str>
                <str name="field">suggest_prefix</str>
                <str name="buildOnOptimize">true</str>
                <str name="buildOnCommit">true</str>
                <str name="buildOnStartup">true</str>
        </lst>

  </searchComponent>


This component works without issued both in standalone and cloud mode, when used as the sole component in a handler, such as in the following excerpt:

        <requestHandler name="/suggest" class="solr.SearchHandler" startup="lazy">
                <lst name="defaults">
                    <str name="suggest.dictionary">default</str>
                    <str name="suggest.dictionary">suggest_phrase</str>
                    <str name="suggest.dictionary">suggest_infix_shingle</str>
                    <str name="suggest.dictionary">suggest_prefix</str>
                    <str name="suggest">true</str>
                    <str name="suggest.count">10</str>
                    <str name="suggest.highlight">false</str>
                </lst>
                <arr name="components">
                        <str>suggest</str>
                </arr>
        </requestHandler>


It also works when used along with other component in standalone mode, such as in the following excerpt, where we use the suggest component to get suggestions during a "normal" search call:

        <requestHandler name="/select" class="solr.SearchHandler">
                <lst name="defaults">
                        <str name="echoParams">explicit</str>
                        <int name="rows">10</int>
                        <str name="df">text_search</str>

                        <str name="defType">edismax</str>

                        <str name="qf">title^5.0 subtitle^3.0 abstract^2.0 text_search</str>
                        <str name="pf">title^5.0 subtitle^3.0 abstract^2.0 text_search</str>
                        <str name="ps">4</str>
                        <str name="spellcheck">on</str>
                        <str name="spellcheck.dictionary">default</str>
                        <str name="spellcheck.extendedResults">true</str>
                        <str name="spellcheck.count">10</str>
                        <str name="spellcheck.alternativeTermCount">5</str>
                        <str name="spellcheck.maxResultsForSuggest">5</str>
                        <str name="spellcheck.collate">true</str>
                        <str name="spellcheck.collateExtendedResults">true</str>
                        <str name="spellcheck.maxCollationTries">10</str>
                        <str name="spellcheck.maxCollations">5</str>

                        <str name="suggest.dictionary">default</str>
                        <str name="suggest.dictionary">suggest_phrase</str>
                        <str name="suggest.dictionary">suggest_infix_shingle</str>
                        <str name="suggest.dictionary">suggest_prefix</str>
                        <str name="suggest">true</str>
                        <str name="suggest.count">10</str>
                        <str name="suggest.highlight">false</str>
                </lst>

                <arr name="last-components">
                        <str>suggest</str>
                        <str>spellcheck</str>
                </arr>
        </requestHandler>

However, the above configuration does not work in cloud mode, where we get an NPE if a search call is made:

 o.a.s.s.HttpSolrCall null:java.lang.NullPointerException
    at org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1034)
    at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:885)
    at org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:585)
    at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:564)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:426)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
    at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
    at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.Server.handle(Server.java:531)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
    at java.lang.Thread.run(Thread.java:748)


As far as we can tell, this is due to the fact that the failing code in QueryComponent assumes that the shard response always contains a 'sort_values' section:

        NamedList sortFieldValues = (NamedList)(srsp.getSolrResponse().getResponse().get("sort_values"));
        NamedList unmarshalledSortFieldValues = unmarshalSortValues(ss, sortFieldValues, schema);

(the exception occurs in unmarshalSortValues(), which assumes the sortFieldValues parameter is not null)

Are we doing something wrong that is preventing 'sort_values' from being present in the shard response? Or is their absence OK, but the code fails to account for it? Our guess is the second, because umarshalSortValues() does check if the values are empty or not, and handles that.

Can anyone help us with this issue? Thanks in advance for any pointers!

Alex

[1] http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201711.mbox/%3CCACKWnqOv=HFQwTxd7SwEqSTj-im+43FE+c9_zbTCPMWsp8vL_w@...%3E

[2] https://issues.apache.org/jira/browse/SOLR-12060



--

Dr. Alexandros Paramythis, M.Sc.
CEO

CONTEXITY AG
Technoparkstrasse 2
CH-8406 Winterthur

T  +41 78 611 6542
E  [hidden email]
W www.contexity.ch

Reply | Threaded
Open this post in threaded view
|

Re: Problems using a suggester component in the /select handler in cloud mode

Jason Gerlowski
Hi Alexandros,

The first step would be to package up your changes in a patch file, and upload that to the JIRA you linked to in your initial email. (SOLR-12060).  More detailed instructions can be found here: https://cwiki.apache.org/confluence/display/solr/HowToContribute#HowToContribute-Generatingapatch.  If you prefer, you can also create a PR on github with your changes, and put a link to the PR on the JIRA ticket.  (There are many guides out there on creating Github PRs, so I won't get into that.)

Thanks for putting in the effort to share your work.  Good luck!

Best,

Jason

On Tue, Jul 16, 2019 at 8:50 AM Alexandros Paramythis <[hidden email]> wrote:
Hi everyone,

We have a fix for the problem described in the message below. Could anyone provide pointers to documentation on how we would go about contributing this back?

Thanks in advance for your input,

Alex


On 26/06/2019 10:48, Alexandros Paramythis wrote:
Hi everyone,

Environment:

Solr 7.5.0, cloud mode (but problem should be identical in multiple versions, at least in 7.x)

Summary:

We have a Solr configuration that returns suggestions in the course of a normal search call (i.e., we have a 'suggest' component added to the 'last-components' for '/select' request handler). This does not work in cloud mode, where we get an NPE in QueryComponent. This problem seems to have been reported in various forms in the past -- see for example [1] and [2] (links at the end of this email) -- but we couldn't find any resolution (or in-depth discussion for that matter).

In more detail:

We have a suggest component configured as follows:

  <searchComponent name="suggest" class="solr.SuggestComponent" >

        <lst name="suggester">
                <str name="name">default</str>
                <str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
                <str name="lookupImpl">org.apache.solr.spelling.suggest.fst.AnalyzingLookupFactory</str>
                <str name="storeDir">dict_default</str>
                <str name="suggestAnalyzerFieldType">text_suggest</str>
                <str name="field">text_suggest</str>
                <str name="buildOnStartup">true</str>
                <str name="buildOnCommit">true</str>
                <str name="buildOnOptimize">true</str>
        </lst>

        <lst name="suggester">
                <str name="name">suggest_phrase</str>
                <str name="lookupImpl">org.apache.solr.spelling.suggest.fst.AnalyzingLookupFactory</str>
                <str name="storeDir">dict_suggest_phrase</str>
                <str name="suggestAnalyzerFieldType">text_suggest_phrase</str>
                <str name="field">suggest_phrase</str>
                <str name="buildOnStartup">true</str>
                <str name="buildOnCommit">true</str>
                <str name="buildOnOptimize">true</str>
        </lst>

        <lst name="suggester">
                <str name="name">suggest_infix_shingle</str>
                <str name="lookupImpl">AnalyzingInfixLookupFactory</str>
                <str name="indexPath">suggestInfixShingleDir</str>
                <str name="suggestAnalyzerFieldType">text_suggest_phrase</str>
                <str name="field">suggest_phrase</str>
                <str name="buildOnOptimize">true</str>
                <str name="buildOnCommit">true</str>
                <str name="buildOnStartup">true</str>
        </lst>

        <lst name="suggester">
                <str name="name">suggest_prefix</str>
                <str name="classname">Suggester</str>
                <str name="lookupImpl">AnalyzingLookupFactory</str>
                <str name="suggestAnalyzerFieldType">text_suggest_prefix</str>
                <str name="field">suggest_prefix</str>
                <str name="buildOnOptimize">true</str>
                <str name="buildOnCommit">true</str>
                <str name="buildOnStartup">true</str>
        </lst>

  </searchComponent>


This component works without issued both in standalone and cloud mode, when used as the sole component in a handler, such as in the following excerpt:

        <requestHandler name="/suggest" class="solr.SearchHandler" startup="lazy">
                <lst name="defaults">
                    <str name="suggest.dictionary">default</str>
                    <str name="suggest.dictionary">suggest_phrase</str>
                    <str name="suggest.dictionary">suggest_infix_shingle</str>
                    <str name="suggest.dictionary">suggest_prefix</str>
                    <str name="suggest">true</str>
                    <str name="suggest.count">10</str>
                    <str name="suggest.highlight">false</str>
                </lst>
                <arr name="components">
                        <str>suggest</str>
                </arr>
        </requestHandler>


It also works when used along with other component in standalone mode, such as in the following excerpt, where we use the suggest component to get suggestions during a "normal" search call:

        <requestHandler name="/select" class="solr.SearchHandler">
                <lst name="defaults">
                        <str name="echoParams">explicit</str>
                        <int name="rows">10</int>
                        <str name="df">text_search</str>

                        <str name="defType">edismax</str>

                        <str name="qf">title^5.0 subtitle^3.0 abstract^2.0 text_search</str>
                        <str name="pf">title^5.0 subtitle^3.0 abstract^2.0 text_search</str>
                        <str name="ps">4</str>
                        <str name="spellcheck">on</str>
                        <str name="spellcheck.dictionary">default</str>
                        <str name="spellcheck.extendedResults">true</str>
                        <str name="spellcheck.count">10</str>
                        <str name="spellcheck.alternativeTermCount">5</str>
                        <str name="spellcheck.maxResultsForSuggest">5</str>
                        <str name="spellcheck.collate">true</str>
                        <str name="spellcheck.collateExtendedResults">true</str>
                        <str name="spellcheck.maxCollationTries">10</str>
                        <str name="spellcheck.maxCollations">5</str>

                        <str name="suggest.dictionary">default</str>
                        <str name="suggest.dictionary">suggest_phrase</str>
                        <str name="suggest.dictionary">suggest_infix_shingle</str>
                        <str name="suggest.dictionary">suggest_prefix</str>
                        <str name="suggest">true</str>
                        <str name="suggest.count">10</str>
                        <str name="suggest.highlight">false</str>
                </lst>

                <arr name="last-components">
                        <str>suggest</str>
                        <str>spellcheck</str>
                </arr>
        </requestHandler>

However, the above configuration does not work in cloud mode, where we get an NPE if a search call is made:

 o.a.s.s.HttpSolrCall null:java.lang.NullPointerException
    at org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1034)
    at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:885)
    at org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:585)
    at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:564)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:426)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
    at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
    at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.Server.handle(Server.java:531)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
    at java.lang.Thread.run(Thread.java:748)


As far as we can tell, this is due to the fact that the failing code in QueryComponent assumes that the shard response always contains a 'sort_values' section:

        NamedList sortFieldValues = (NamedList)(srsp.getSolrResponse().getResponse().get("sort_values"));
        NamedList unmarshalledSortFieldValues = unmarshalSortValues(ss, sortFieldValues, schema);

(the exception occurs in unmarshalSortValues(), which assumes the sortFieldValues parameter is not null)

Are we doing something wrong that is preventing 'sort_values' from being present in the shard response? Or is their absence OK, but the code fails to account for it? Our guess is the second, because umarshalSortValues() does check if the values are empty or not, and handles that.

Can anyone help us with this issue? Thanks in advance for any pointers!

Alex

[1] http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201711.mbox/%3CCACKWnqOv=HFQwTxd7SwEqSTj-im+43FE+c9_zbTCPMWsp8vL_w@...%3E

[2] https://issues.apache.org/jira/browse/SOLR-12060



--

Dr. Alexandros Paramythis, M.Sc.
CEO

CONTEXITY AG
Technoparkstrasse 2
CH-8406 Winterthur

T  +41 78 611 6542
E  [hidden email]
W www.contexity.ch