Solr SynonymGraphFilterFactory error on import

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr SynonymGraphFilterFactory error on import

damian.pawski
After upgrading to Solr 7.2 import started to log errors for some documents.

Field that returns errors:

   <fieldType name="text" class="solr.TextField" positionIncrementGap="100">
  <analyzer type="index">
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
    <filter class="solr.FlattenGraphFilterFactory"/>
    <filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" />
    <filter class="solr.WordDelimiterGraphFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"
preserveOriginal="1"/>
    <filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
    <filter class="solr.ASCIIFoldingFilterFactory"/>
    <filter class="solr.EnglishMinimalStemFilterFactory"/>
    <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt"/>
    <filter class="solr.WordDelimiterGraphFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="0"
catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
    <filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
    <filter class="solr.ASCIIFoldingFilterFactory"/>
    <filter class="solr.EnglishMinimalStemFilterFactory"/>
    <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
  </analyzer>
</fieldType>
During the import below error is returned for some of the records:

org.apache.solr.common.SolrException: Exception writing document id XXXXX to
the index; possible analysis error: startOffset must be non-negative, and
endOffset must be >= startOffset, and offsets must not go backwards
startOffset=2874,endOffset=2878,lastStartOffset=2879 for field 'XXXXX'
at
g.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:226)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:936)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:616)
at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:80)


It is related to the:
<filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
<filter class="solr.FlattenGraphFilterFactory"/>

If I remove this it works fine, previously we were using:
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>

and it was working fine, but the SynonymFilterFactory is not longer
supported on the Solr 7.X., it has been replaced with
SynonymGraphFilterFactory, I have added FlattenGraphFilterFactory as
suggested.

I am not sure why Solr returns those errors?

Thank you in advance for suggestions.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr SynonymGraphFilterFactory error on import

WebsterHomer
You probably want to call  solr.FlattenGraphFilterFactory  after the call
to  WordDelimiterGraphFilterFactory. I put it at the end

Also there is an issue calling more than one graph filter in an analysis
chain so you may need to remove one of them. I think that there is a Jira
about that

Personally I prefer to do synonyms at query time.



On Mon, Mar 5, 2018 at 3:56 AM, damian.pawski <[hidden email]> wrote:

> After upgrading to Solr 7.2 import started to log errors for some
> documents.
>
> Field that returns errors:
>
>    <fieldType name="text" class="solr.TextField"
> positionIncrementGap="100">
>   <analyzer type="index">
>     <tokenizer class="solr.StandardTokenizerFactory"/>
>     <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt"
> ignoreCase="true" expand="true"/>
>     <filter class="solr.FlattenGraphFilterFactory"/>
>     <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt" />
>     <filter class="solr.WordDelimiterGraphFilterFactory"
> generateWordParts="1" generateNumberParts="1" catenateWords="1"
> catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"
> preserveOriginal="1"/>
>     <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.EnglishPossessiveFilterFactory"/>
>     <filter class="solr.ASCIIFoldingFilterFactory"/>
>     <filter class="solr.EnglishMinimalStemFilterFactory"/>
>     <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>   </analyzer>
>   <analyzer type="query">
>     <tokenizer class="solr.StandardTokenizerFactory"/>
>     <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt"/>
>     <filter class="solr.WordDelimiterGraphFilterFactory"
> generateWordParts="1" generateNumberParts="1" catenateWords="0"
> catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
>     <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.EnglishPossessiveFilterFactory"/>
>     <filter class="solr.ASCIIFoldingFilterFactory"/>
>     <filter class="solr.EnglishMinimalStemFilterFactory"/>
>     <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>   </analyzer>
> </fieldType>
> During the import below error is returned for some of the records:
>
> org.apache.solr.common.SolrException: Exception writing document id XXXXX
> to
> the index; possible analysis error: startOffset must be non-negative, and
> endOffset must be >= startOffset, and offsets must not go backwards
> startOffset=2874,endOffset=2878,lastStartOffset=2879 for field 'XXXXX'
> at
> g.apache.solr.update.DirectUpdateHandler2.addDoc(
> DirectUpdateHandler2.java:226)
> at
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(
> RunUpdateProcessorFactory.java:67)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(
> UpdateRequestProcessor.java:55)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(
> DistributedUpdateProcessor.java:936)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(
> DistributedUpdateProcessor.java:616)
> at
> org.apache.solr.update.processor.LogUpdateProcessorFactory$
> LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> at org.apache.solr.handler.dataimport.SolrWriter.upload(
> SolrWriter.java:80)
>
>
> It is related to the:
> <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt"
> ignoreCase="true" expand="true"/>
> <filter class="solr.FlattenGraphFilterFactory"/>
>
> If I remove this it works fine, previously we were using:
> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
> ignoreCase="true" expand="true"/>
>
> and it was working fine, but the SynonymFilterFactory is not longer
> supported on the Solr 7.X., it has been replaced with
> SynonymGraphFilterFactory, I have added FlattenGraphFilterFactory as
> suggested.
>
> I am not sure why Solr returns those errors?
>
> Thank you in advance for suggestions.
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>

--


This message and any attachment are confidential and may be privileged or
otherwise protected from disclosure. If you are not the intended recipient,
you must not copy this message or attachment or disclose the contents to
any other person. If you have received this transmission in error, please
notify the sender immediately and delete the message and any attachment
from your system. Merck KGaA, Darmstadt, Germany and any of its
subsidiaries do not accept liability for any omissions or errors in this
message which may arise as a result of E-Mail-transmission or for damages
resulting from any unauthorized changes of the content of this message and
any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its
subsidiaries do not guarantee that this message is free of viruses and does
not accept liability for any damages caused by any virus transmitted
therewith.

Click http://www.emdgroup.com/disclaimer to access the German, French,
Spanish and Portuguese versions of this disclaimer.
Reply | Threaded
Open this post in threaded view
|

Re: Solr SynonymGraphFilterFactory error on import

damian.pawski
"/You probably want to call  solr.FlattenGraphFilterFactory  after the call
to  WordDelimiterGraphFilterFactory. I put it at the end/ "  

That solved my issue

Thank you



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html