Solr 7.2.1 - unexpected docvalues type

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr 7.2.1 - unexpected docvalues type

Antony Alphonse
Hi,

I shared the collection and re-indexed the data with the same schema. But
one of the field is throwing the below error. Any suggestions?


<field name="lowercase" type="lowercase" indexed="true" stored="true"
required="false" />


ERROR (qtp672320506-32) [c: s:shard3 r:core_node01 x:_shard3_replica_n69]
o.a.s.h.RequestHandlerBase java.lang.IllegalStateException: unexpected
docvalues type SORTED_SET for field 'lowercase' (expected=SORTED). Re-index
with correct docvalues type.
        at org.apache.lucene.index.DocValues.checkField(DocValues.java:340)
        at org.apache.lucene.index.DocValues.getSorted(DocValues.java:392)
        at
org.apache.lucene.search.grouping.TermGroupSelector.setNextReader(TermGroupSelector.java:56)
        at
org.apache.lucene.search.grouping.FirstPassGroupingCollector.doSetNextReader(FirstPassGroupingCollector.java:350)
        at
org.apache.lucene.search.SimpleCollector.getLeafCollector(SimpleCollector.java:33)
        at
org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:121)
        at
org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:121)
        at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:651)
        at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
        at
org.apache.solr.search.grouping.CommandHandler.searchWithTimeLimiter(CommandHandler.java:239)
        at
org.apache.solr.search.grouping.CommandHandler.execute(CommandHandler.java:162)
        at
org.apache.solr.handler.component.QueryComponent.doProcessGroupedDistributedSearchFirstPhase(QueryComponent.java:1279)
        at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:360)
        at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
        at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)
        at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
        at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
        at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
        at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
        at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:745)

Thanks!!
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.2.1 - unexpected docvalues type

Shawn Heisey-2
On 11/8/2019 5:31 PM, Antony Alphonse wrote:

> I shared the collection and re-indexed the data with the same schema. But
> one of the field is throwing the below error. Any suggestions?
>
> <field name="lowercase" type="lowercase" indexed="true" stored="true"
> required="false" />
>
> ERROR (qtp672320506-32) [c: s:shard3 r:core_node01 x:_shard3_replica_n69]
> o.a.s.h.RequestHandlerBase java.lang.IllegalStateException: unexpected
> docvalues type SORTED_SET for field 'lowercase' (expected=SORTED). Re-index
> with correct docvalues type.

This error means that part of the index was created with one definition
for the field in question, then the schema was changed in an
incompatible way, and additional indexing was attempted.

The solution to this particular error is to completely delete the index
directories that make up the collection, reload it, and then build it
from scratch again.  The error happens at the Lucene level and the only
way to fix it is to completely delete the index.  You could do it by
creating an entirely new collection.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.2.1 - unexpected docvalues type

Antony Alphonse
>
> Hi Shawn,
>

I will try that solution. Also I had to mention that the queries that fail
with this error has the "group.field":"lowercase". Should I change the
field type?

Thanks,
Antony
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.2.1 - unexpected docvalues type

Erick Erickson
We can’t answer whether you should change the field type for two reasons:

1> It depends on your use case.
2> we don’t know what the field type “lowercase” does. It’s composed of an analysis chain that you may have changed. And whatever config you are using may have changed with different releases of Solr.

Grouping is generally done on a docValues-eligible field type. AFAIK, “lowercase” is a solr-text based field so is ineligible for docValues. I’ve got to guess here, but I’d suggest you start with a fieldType of “string”, and enable docValues on it.

Best,
Erick



> On Nov 9, 2019, at 12:54 AM, Antony Alphonse <[hidden email]> wrote:
>
>>
>> Hi Shawn,
>>
>
> I will try that solution. Also I had to mention that the queries that fail
> with this error has the "group.field":"lowercase". Should I change the
> field type?
>
> Thanks,
> Antony

Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.2.1 - unexpected docvalues type

Antony Alphonse
Hi Shawn,

Thank you. I switched the fieldType=string and it worked. I might have to
check on the use-case to see if "string" will work for us.

I have noted the "lowercase" field type which I believe is similar to the
one in schema ver 1.6.


              <fieldType name="lowercase" class="solr.TextField"
                        positionIncrementGap="100">
                        <analyzer>
                                <tokenizer
class="solr.KeywordTokenizerFactory" />
                                <filter class="solr.LowerCaseFilterFactory"
/>
                        </analyzer>
                </fieldType>

Thanks,
Antony

On Sat, Nov 9, 2019 at 7:52 AM Erick Erickson <[hidden email]>
wrote:

> We can’t answer whether you should change the field type for two reasons:
>
> 1> It depends on your use case.
> 2> we don’t know what the field type “lowercase” does. It’s composed of an
> analysis chain that you may have changed. And whatever config you are using
> may have changed with different releases of Solr.
>
> Grouping is generally done on a docValues-eligible field type. AFAIK,
> “lowercase” is a solr-text based field so is ineligible for docValues. I’ve
> got to guess here, but I’d suggest you start with a fieldType of “string”,
> and enable docValues on it.
>
> Best,
> Erick
>
>
>
> > On Nov 9, 2019, at 12:54 AM, Antony Alphonse <[hidden email]>
> wrote:
> >
> >>
> >> Hi Shawn,
> >>
> >
> > I will try that solution. Also I had to mention that the queries that
> fail
> > with this error has the "group.field":"lowercase". Should I change the
> > field type?
> >
> > Thanks,
> > Antony
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.2.1 - unexpected docvalues type

Erick Erickson
So “lowercase” is, indeed, a solr.TextField, which is ineligible for docValues. Given that definition, the difference will be that a “string” type is totally un-analyzed, so the values that go into the index and the query itself will be case-sensitive. You’ll have to pre-process both to do the right thing.

> On Nov 9, 2019, at 6:15 PM, Antony Alphonse <[hidden email]> wrote:
>
> Hi Shawn,
>
> Thank you. I switched the fieldType=string and it worked. I might have to
> check on the use-case to see if "string" will work for us.
>
> I have noted the "lowercase" field type which I believe is similar to the
> one in schema ver 1.6.
>
>
>              <fieldType name="lowercase" class="solr.TextField"
>                        positionIncrementGap="100">
>                        <analyzer>
>                                <tokenizer
> class="solr.KeywordTokenizerFactory" />
>                                <filter class="solr.LowerCaseFilterFactory"
> />
>                        </analyzer>
>                </fieldType>
>
> Thanks,
> Antony
>
> On Sat, Nov 9, 2019 at 7:52 AM Erick Erickson <[hidden email]>
> wrote:
>
>> We can’t answer whether you should change the field type for two reasons:
>>
>> 1> It depends on your use case.
>> 2> we don’t know what the field type “lowercase” does. It’s composed of an
>> analysis chain that you may have changed. And whatever config you are using
>> may have changed with different releases of Solr.
>>
>> Grouping is generally done on a docValues-eligible field type. AFAIK,
>> “lowercase” is a solr-text based field so is ineligible for docValues. I’ve
>> got to guess here, but I’d suggest you start with a fieldType of “string”,
>> and enable docValues on it.
>>
>> Best,
>> Erick
>>
>>
>>
>>> On Nov 9, 2019, at 12:54 AM, Antony Alphonse <[hidden email]>
>> wrote:
>>>
>>>>
>>>> Hi Shawn,
>>>>
>>>
>>> I will try that solution. Also I had to mention that the queries that
>> fail
>>> with this error has the "group.field":"lowercase". Should I change the
>>> field type?
>>>
>>> Thanks,
>>> Antony
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.2.1 - unexpected docvalues type

Emir Arnautović
Hi Antony,
Like Erick explained, you still have to preprocess your field in order to be able to use doc values. What you can do is use update request processor chain and have all the logic in Solr. Here is blog post explaining how it could work: https://www.od-bits.com/2018/02/solr-docvalues-on-analysed-field.html <https://www.od-bits.com/2018/02/solr-docvalues-on-analysed-field.html>

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 10 Nov 2019, at 15:54, Erick Erickson <[hidden email]> wrote:
>
> So “lowercase” is, indeed, a solr.TextField, which is ineligible for docValues. Given that definition, the difference will be that a “string” type is totally un-analyzed, so the values that go into the index and the query itself will be case-sensitive. You’ll have to pre-process both to do the right thing.
>
>> On Nov 9, 2019, at 6:15 PM, Antony Alphonse <[hidden email]> wrote:
>>
>> Hi Shawn,
>>
>> Thank you. I switched the fieldType=string and it worked. I might have to
>> check on the use-case to see if "string" will work for us.
>>
>> I have noted the "lowercase" field type which I believe is similar to the
>> one in schema ver 1.6.
>>
>>
>>             <fieldType name="lowercase" class="solr.TextField"
>>                       positionIncrementGap="100">
>>                       <analyzer>
>>                               <tokenizer
>> class="solr.KeywordTokenizerFactory" />
>>                               <filter class="solr.LowerCaseFilterFactory"
>> />
>>                       </analyzer>
>>               </fieldType>
>>
>> Thanks,
>> Antony
>>
>> On Sat, Nov 9, 2019 at 7:52 AM Erick Erickson <[hidden email]>
>> wrote:
>>
>>> We can’t answer whether you should change the field type for two reasons:
>>>
>>> 1> It depends on your use case.
>>> 2> we don’t know what the field type “lowercase” does. It’s composed of an
>>> analysis chain that you may have changed. And whatever config you are using
>>> may have changed with different releases of Solr.
>>>
>>> Grouping is generally done on a docValues-eligible field type. AFAIK,
>>> “lowercase” is a solr-text based field so is ineligible for docValues. I’ve
>>> got to guess here, but I’d suggest you start with a fieldType of “string”,
>>> and enable docValues on it.
>>>
>>> Best,
>>> Erick
>>>
>>>
>>>
>>>> On Nov 9, 2019, at 12:54 AM, Antony Alphonse <[hidden email]>
>>> wrote:
>>>>
>>>>>
>>>>> Hi Shawn,
>>>>>
>>>>
>>>> I will try that solution. Also I had to mention that the queries that
>>> fail
>>>> with this error has the "group.field":"lowercase". Should I change the
>>>> field type?
>>>>
>>>> Thanks,
>>>> Antony
>>>
>>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.2.1 - unexpected docvalues type

Antony Alphonse
Thank you both. I will look into the options.

-AA

On Mon, Nov 11, 2019 at 6:05 AM Emir Arnautović <
[hidden email]> wrote:

> Hi Antony,
> Like Erick explained, you still have to preprocess your field in order to
> be able to use doc values. What you can do is use update request processor
> chain and have all the logic in Solr. Here is blog post explaining how it
> could work:
> https://www.od-bits.com/2018/02/solr-docvalues-on-analysed-field.html <
> https://www.od-bits.com/2018/02/solr-docvalues-on-analysed-field.html>
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
> > On 10 Nov 2019, at 15:54, Erick Erickson <[hidden email]>
> wrote:
> >
> > So “lowercase” is, indeed, a solr.TextField, which is ineligible for
> docValues. Given that definition, the difference will be that a “string”
> type is totally un-analyzed, so the values that go into the index and the
> query itself will be case-sensitive. You’ll have to pre-process both to do
> the right thing.
> >
> >> On Nov 9, 2019, at 6:15 PM, Antony Alphonse <[hidden email]>
> wrote:
> >>
> >> Hi Shawn,
> >>
> >> Thank you. I switched the fieldType=string and it worked. I might have
> to
> >> check on the use-case to see if "string" will work for us.
> >>
> >> I have noted the "lowercase" field type which I believe is similar to
> the
> >> one in schema ver 1.6.
> >>
> >>
> >>             <fieldType name="lowercase" class="solr.TextField"
> >>                       positionIncrementGap="100">
> >>                       <analyzer>
> >>                               <tokenizer
> >> class="solr.KeywordTokenizerFactory" />
> >>                               <filter
> class="solr.LowerCaseFilterFactory"
> >> />
> >>                       </analyzer>
> >>               </fieldType>
> >>
> >> Thanks,
> >> Antony
> >>
> >> On Sat, Nov 9, 2019 at 7:52 AM Erick Erickson <[hidden email]>
> >> wrote:
> >>
> >>> We can’t answer whether you should change the field type for two
> reasons:
> >>>
> >>> 1> It depends on your use case.
> >>> 2> we don’t know what the field type “lowercase” does. It’s composed
> of an
> >>> analysis chain that you may have changed. And whatever config you are
> using
> >>> may have changed with different releases of Solr.
> >>>
> >>> Grouping is generally done on a docValues-eligible field type. AFAIK,
> >>> “lowercase” is a solr-text based field so is ineligible for docValues.
> I’ve
> >>> got to guess here, but I’d suggest you start with a fieldType of
> “string”,
> >>> and enable docValues on it.
> >>>
> >>> Best,
> >>> Erick
> >>>
> >>>
> >>>
> >>>> On Nov 9, 2019, at 12:54 AM, Antony Alphonse <
> [hidden email]>
> >>> wrote:
> >>>>
> >>>>>
> >>>>> Hi Shawn,
> >>>>>
> >>>>
> >>>> I will try that solution. Also I had to mention that the queries that
> >>> fail
> >>>> with this error has the "group.field":"lowercase". Should I change the
> >>>> field type?
> >>>>
> >>>> Thanks,
> >>>> Antony
> >>>
> >>>
> >
>
>