Consuming token stream more than once in same filter

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Consuming token stream more than once in same filter

sjhyam
I have a requirement to handle synonyms differently based on the first word (token) in the text field of the document. I have implemented custom SynFilterFactory which loads synonyms per languages when core/solr is started.

Now in the MySynonymFilterFactory#create(TokenStream input) method, I have to read the first token from the input TokenStream. Based on that token value, corresponding SynonymMap will be used for SynonymFilter creation.

Here are my documents
doc1 <text>lang_eng this is English language text</text>
doc2 <text>lang_fra this is French language text</text>
doc3 <text>lang_spa this is Spanish language text</text>

MySynonymFilterFactory creates MySynonymFilter. Method create() logic is below...

@Override

public TokenStream create(TokenStream input) {

// if the fst is null, it means there's actually no synonyms... just return the

// original stream as there is nothing to do here.

// return map.fst == null ? input : new MySynonymFilter(input, map, ignoreCase);

System.out.println("input=" + input);

// some how read the TokenStream here to capture the lang value

SynonymMap synonyms = null;

try {

CharTermAttribute termAtt = input.addAttribute(CharTermAttribute.class);

boolean first = false;

input.reset();

while (!first && input.incrementToken()) {

String term = new String(termAtt.buffer(), 0, termAtt.length());

System.out.println("termAtt=" + term);

if (StringUtils.startsWith(term, "lang_")) {

String[] split = StringUtils.split(term, "_");

String lang = split[1];

String key = (langSynMap.containsKey(lang)) ? lang : "generic";

synonyms = langSynMap.get(key);

System.out.println("synonyms=" + synonyms);

}

first = true;

}

} catch (IOException e) {

// TODO Auto-generated catch block

e.printStackTrace();

}


return synonyms == null ? input : new SynonymFilter(input, synonyms, ignoreCase);

}


This code compiles and this new analysis works fine in the Solr admin analysis screen. But same fails with below exception when I try to index a document
30273 ERROR (qtp1689843956-18) [   x:gcom] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Exception writing document id id1 to the index; possible analysis error.
        at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:180)
        at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:934)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1089)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:712)
        at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
Caused by: java.lang.IllegalStateException: TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Java
docs of TokenStream class for more information about the correct consuming workflow.
        at org.apache.lucene.analysis.Tokenizer$1.read(Tokenizer.java:109)
        at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:527)
        at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:738)
        at org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:159)
        at com.synonyms.poc.synpoc.MySynonymFilterFactory.create(MySynonymFilterFactory.java:94)
        at org.apache.solr.analysis.TokenizerChain.createComponents(TokenizerChain.java:91)
        at org.apache.lucene.analysis.AnalyzerWrapper.createComponents(AnalyzerWrapper.java:101)
        at org.apache.lucene.analysis.AnalyzerWrapper.createComponents(AnalyzerWrapper.java:101)
        at org.apache.lucene.analysis.Analyzer.tokenStream(Analyzer.java:176)
        at org.apache.lucene.document.Field.tokenStream(Field.java:562)
        at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:628)
        at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:365)
        at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:321)
        at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
        at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:282)
        at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:214)
        at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:169)
        ... 37 more

Any idea how can I read a token stream with out violating the token stream contract. I see a similar discussion here https://lucene.472066.n3.nabble.com/how-to-reuse-a-tokenStream-td850767.html, but doesn't help solve my problem.

Also how come same error is not reported when analyzing the field value using Solr admin console analysis screen.

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: Consuming token stream more than once in same filter

Michael Sokolov-4
Are you able to:
1) create a custom attribute encoding the language
2) create a filter that sets the attribute when it reads the first token
3) wrap your synonym filters (one for each language) in a
ConditionalTokenFilter that filters based on the language attribute

On Wed, Oct 30, 2019 at 11:16 PM Shyamsunder Mutcha <[hidden email]> wrote:

>
> I have a requirement to handle synonyms differently based on the first word (token) in the text field of the document. I have implemented custom SynFilterFactory which loads synonyms per languages when core/solr is started.
>
> Now in the MySynonymFilterFactory#create(TokenStream input) method, I have to read the first token from the input TokenStream. Based on that token value, corresponding SynonymMap will be used for SynonymFilter creation.
>
> Here are my documents
> doc1 <text>lang_eng this is English language text</text>
> doc2 <text>lang_fra this is French language text</text>
> doc3 <text>lang_spa this is Spanish language text</text>
>
> MySynonymFilterFactory creates MySynonymFilter. Method create() logic is below...
>
> @Override
>
> public TokenStream create(TokenStream input) {
>
> // if the fst is null, it means there's actually no synonyms... just return the
>
> // original stream as there is nothing to do here.
>
> // return map.fst == null ? input : new MySynonymFilter(input, map, ignoreCase);
>
> System.out.println("input=" + input);
>
> // some how read the TokenStream here to capture the lang value
>
> SynonymMap synonyms = null;
>
> try {
>
> CharTermAttribute termAtt = input.addAttribute(CharTermAttribute.class);
>
> boolean first = false;
>
> input.reset();
>
> while (!first && input.incrementToken()) {
>
> String term = new String(termAtt.buffer(), 0, termAtt.length());
>
> System.out.println("termAtt=" + term);
>
> if (StringUtils.startsWith(term, "lang_")) {
>
> String[] split = StringUtils.split(term, "_");
>
> String lang = split[1];
>
> String key = (langSynMap.containsKey(lang)) ? lang : "generic";
>
> synonyms = langSynMap.get(key);
>
> System.out.println("synonyms=" + synonyms);
>
> }
>
> first = true;
>
> }
>
> } catch (IOException e) {
>
> // TODO Auto-generated catch block
>
> e.printStackTrace();
>
> }
>
>
> return synonyms == null ? input : new SynonymFilter(input, synonyms, ignoreCase);
>
> }
>
>
> This code compiles and this new analysis works fine in the Solr admin analysis screen. But same fails with below exception when I try to index a document
> 30273 ERROR (qtp1689843956-18) [   x:gcom] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Exception writing document id id1 to the index; possible analysis error.
>         at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:180)
>         at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
>         at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
>         at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:934)
>         at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1089)
>         at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:712)
>         at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> Caused by: java.lang.IllegalStateException: TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Java
> docs of TokenStream class for more information about the correct consuming workflow.
>         at org.apache.lucene.analysis.Tokenizer$1.read(Tokenizer.java:109)
>         at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:527)
>         at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:738)
>         at org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:159)
>         at com.synonyms.poc.synpoc.MySynonymFilterFactory.create(MySynonymFilterFactory.java:94)
>         at org.apache.solr.analysis.TokenizerChain.createComponents(TokenizerChain.java:91)
>         at org.apache.lucene.analysis.AnalyzerWrapper.createComponents(AnalyzerWrapper.java:101)
>         at org.apache.lucene.analysis.AnalyzerWrapper.createComponents(AnalyzerWrapper.java:101)
>         at org.apache.lucene.analysis.Analyzer.tokenStream(Analyzer.java:176)
>         at org.apache.lucene.document.Field.tokenStream(Field.java:562)
>         at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:628)
>         at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:365)
>         at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:321)
>         at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
>         at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
>         at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:282)
>         at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:214)
>         at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:169)
>         ... 37 more
>
> Any idea how can I read a token stream with out violating the token stream contract. I see a similar discussion here https://lucene.472066.n3.nabble.com/how-to-reuse-a-tokenStream-td850767.html, but doesn't help solve my problem.
>
> Also how come same error is not reported when analyzing the field value using Solr admin console analysis screen.
>
> Thanks

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Consuming token stream more than once in same filter

sjhyam
Thanks for your inputs Michael. Looks like ConditionalTokenFilter is introduced from Lucene 7.4 version. I have implemented similar approach where I read the first token from input stream in MySynonymFilterFactory and load the language specific SynonymMap to create MySynonymFilter. This approach seems to be working when I tested the field analysis using Solr admin console analysis page. But same reports a error message when index a document.

When I try to ping the TokenStream input parameter, I see it has two different class types

**Admin console Analysis***
input=ListBasedTokenStream@3a90e594 term=,bytes=[],startOffset=13,endOffset=13,positionIncrement=0,positionLength=1,type=word,position=3,positionHistory=[Ljava.lang.Integer;@34571cb6

**Solr indexing***
input=StandardTokenizer@41448310 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word

I have added a input.reset/close/end method in the constructor of MySynonymFilterFactory, but that didn't work. So I need a way to consume a token stream twice with out breaking the contract
- consume it once to read the first token
- then reset it to original state
- consume it in the SynonymFilter where synonyms are applied as per the first token which is saved as variable in that SynonymFilter

Thanks

On 2019/10/31 08:54:02, Michael Sokolov <[hidden email]> wrote:

> Are you able to:
> 1) create a custom attribute encoding the language
> 2) create a filter that sets the attribute when it reads the first token
> 3) wrap your synonym filters (one for each language) in a
> ConditionalTokenFilter that filters based on the language attribute
>
> On Wed, Oct 30, 2019 at 11:16 PM Shyamsunder Mutcha <[hidden email]> wrote:
> >
> > I have a requirement to handle synonyms differently based on the first word (token) in the text field of the document. I have implemented custom SynFilterFactory which loads synonyms per languages when core/solr is started.
> >
> > Now in the MySynonymFilterFactory#create(TokenStream input) method, I have to read the first token from the input TokenStream. Based on that token value, corresponding SynonymMap will be used for SynonymFilter creation.
> >
> > Here are my documents
> > doc1 <text>lang_eng this is English language text</text>
> > doc2 <text>lang_fra this is French language text</text>
> > doc3 <text>lang_spa this is Spanish language text</text>
> >
> > MySynonymFilterFactory creates MySynonymFilter. Method create() logic is below...
> >
> > @Override
> >
> > public TokenStream create(TokenStream input) {
> >
> > // if the fst is null, it means there's actually no synonyms... just return the
> >
> > // original stream as there is nothing to do here.
> >
> > // return map.fst == null ? input : new MySynonymFilter(input, map, ignoreCase);
> >
> > System.out.println("input=" + input);
> >
> > // some how read the TokenStream here to capture the lang value
> >
> > SynonymMap synonyms = null;
> >
> > try {
> >
> > CharTermAttribute termAtt = input.addAttribute(CharTermAttribute.class);
> >
> > boolean first = false;
> >
> > input.reset();
> >
> > while (!first && input.incrementToken()) {
> >
> > String term = new String(termAtt.buffer(), 0, termAtt.length());
> >
> > System.out.println("termAtt=" + term);
> >
> > if (StringUtils.startsWith(term, "lang_")) {
> >
> > String[] split = StringUtils.split(term, "_");
> >
> > String lang = split[1];
> >
> > String key = (langSynMap.containsKey(lang)) ? lang : "generic";
> >
> > synonyms = langSynMap.get(key);
> >
> > System.out.println("synonyms=" + synonyms);
> >
> > }
> >
> > first = true;
> >
> > }
> >
> > } catch (IOException e) {
> >
> > // TODO Auto-generated catch block
> >
> > e.printStackTrace();
> >
> > }
> >
> >
> > return synonyms == null ? input : new SynonymFilter(input, synonyms, ignoreCase);
> >
> > }
> >
> >
> > This code compiles and this new analysis works fine in the Solr admin analysis screen. But same fails with below exception when I try to index a document
> > 30273 ERROR (qtp1689843956-18) [   x:gcom] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Exception writing document id id1 to the index; possible analysis error.
> >         at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:180)
> >         at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
> >         at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
> >         at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:934)
> >         at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1089)
> >         at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:712)
> >         at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> > Caused by: java.lang.IllegalStateException: TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Java
> > docs of TokenStream class for more information about the correct consuming workflow.
> >         at org.apache.lucene.analysis.Tokenizer$1.read(Tokenizer.java:109)
> >         at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:527)
> >         at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:738)
> >         at org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:159)
> >         at com.synonyms.poc.synpoc.MySynonymFilterFactory.create(MySynonymFilterFactory.java:94)
> >         at org.apache.solr.analysis.TokenizerChain.createComponents(TokenizerChain.java:91)
> >         at org.apache.lucene.analysis.AnalyzerWrapper.createComponents(AnalyzerWrapper.java:101)
> >         at org.apache.lucene.analysis.AnalyzerWrapper.createComponents(AnalyzerWrapper.java:101)
> >         at org.apache.lucene.analysis.Analyzer.tokenStream(Analyzer.java:176)
> >         at org.apache.lucene.document.Field.tokenStream(Field.java:562)
> >         at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:628)
> >         at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:365)
> >         at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:321)
> >         at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
> >         at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
> >         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
> >         at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:282)
> >         at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:214)
> >         at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:169)
> >         ... 37 more
> >
> > Any idea how can I read a token stream with out violating the token stream contract. I see a similar discussion here https://lucene.472066.n3.nabble.com/how-to-reuse-a-tokenStream-td850767.html, but doesn't help solve my problem.
> >
> > Also how come same error is not reported when analyzing the field value using Solr admin console analysis screen.
> >
> > Thanks
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Consuming token stream more than once in same filter

sjhyam
Other way to understand what I am trying to solve here, consider two documents like this

doc1:
<doc>
 <lang>eng</lang>
 <text>English text</text>
<doc>

doc2:
<doc>
 <lang>fra</lang>
 <text>French text</text>
<doc>

So analysis time for text field, is there a way for me know the lang field value. I see that Analyzer/Tokenizer has no reference to other field value. So what I am doing is in the Update Request Processor, I am adding the lang value as first word. So here is what my document looks like after URP

<doc>
 <lang>eng</lang>
 <text>lang_eng English text</text>
<doc>

doc2:
<doc>
 <lang>fra</lang>
 <text>lang_fra French text</text>
<doc>

So in my down stream tokenizer/tokenfilter classes, I plan to read the first token and it starts with lang_ value, then read the language value (eng/fra), and apply corresponding token filter. Also I skip this first token if it starts with "lang_", so this way this token is not added to the index.

Sounds like a crazy problem as I am trying to add all language documents into one single core. If I am creating a core per each language, this problem is solved out of box.

Thanks

On 2019/10/31 10:49:31, Shyamsunder Mutcha <[hidden email]> wrote:

> Thanks for your inputs Michael. Looks like ConditionalTokenFilter is introduced from Lucene 7.4 version. I have implemented similar approach where I read the first token from input stream in MySynonymFilterFactory and load the language specific SynonymMap to create MySynonymFilter. This approach seems to be working when I tested the field analysis using Solr admin console analysis page. But same reports a error message when index a document.
>
> When I try to ping the TokenStream input parameter, I see it has two different class types
>
> **Admin console Analysis***
> input=ListBasedTokenStream@3a90e594 term=,bytes=[],startOffset=13,endOffset=13,positionIncrement=0,positionLength=1,type=word,position=3,positionHistory=[Ljava.lang.Integer;@34571cb6
>
> **Solr indexing***
> input=StandardTokenizer@41448310 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word
>
> I have added a input.reset/close/end method in the constructor of MySynonymFilterFactory, but that didn't work. So I need a way to consume a token stream twice with out breaking the contract
> - consume it once to read the first token
> - then reset it to original state
> - consume it in the SynonymFilter where synonyms are applied as per the first token which is saved as variable in that SynonymFilter
>
> Thanks
>
> On 2019/10/31 08:54:02, Michael Sokolov <[hidden email]> wrote:
> > Are you able to:
> > 1) create a custom attribute encoding the language
> > 2) create a filter that sets the attribute when it reads the first token
> > 3) wrap your synonym filters (one for each language) in a
> > ConditionalTokenFilter that filters based on the language attribute
> >
> > On Wed, Oct 30, 2019 at 11:16 PM Shyamsunder Mutcha <[hidden email]> wrote:
> > >
> > > I have a requirement to handle synonyms differently based on the first word (token) in the text field of the document. I have implemented custom SynFilterFactory which loads synonyms per languages when core/solr is started.
> > >
> > > Now in the MySynonymFilterFactory#create(TokenStream input) method, I have to read the first token from the input TokenStream. Based on that token value, corresponding SynonymMap will be used for SynonymFilter creation.
> > >
> > > Here are my documents
> > > doc1 <text>lang_eng this is English language text</text>
> > > doc2 <text>lang_fra this is French language text</text>
> > > doc3 <text>lang_spa this is Spanish language text</text>
> > >
> > > MySynonymFilterFactory creates MySynonymFilter. Method create() logic is below...
> > >
> > > @Override
> > >
> > > public TokenStream create(TokenStream input) {
> > >
> > > // if the fst is null, it means there's actually no synonyms... just return the
> > >
> > > // original stream as there is nothing to do here.
> > >
> > > // return map.fst == null ? input : new MySynonymFilter(input, map, ignoreCase);
> > >
> > > System.out.println("input=" + input);
> > >
> > > // some how read the TokenStream here to capture the lang value
> > >
> > > SynonymMap synonyms = null;
> > >
> > > try {
> > >
> > > CharTermAttribute termAtt = input.addAttribute(CharTermAttribute.class);
> > >
> > > boolean first = false;
> > >
> > > input.reset();
> > >
> > > while (!first && input.incrementToken()) {
> > >
> > > String term = new String(termAtt.buffer(), 0, termAtt.length());
> > >
> > > System.out.println("termAtt=" + term);
> > >
> > > if (StringUtils.startsWith(term, "lang_")) {
> > >
> > > String[] split = StringUtils.split(term, "_");
> > >
> > > String lang = split[1];
> > >
> > > String key = (langSynMap.containsKey(lang)) ? lang : "generic";
> > >
> > > synonyms = langSynMap.get(key);
> > >
> > > System.out.println("synonyms=" + synonyms);
> > >
> > > }
> > >
> > > first = true;
> > >
> > > }
> > >
> > > } catch (IOException e) {
> > >
> > > // TODO Auto-generated catch block
> > >
> > > e.printStackTrace();
> > >
> > > }
> > >
> > >
> > > return synonyms == null ? input : new SynonymFilter(input, synonyms, ignoreCase);
> > >
> > > }
> > >
> > >
> > > This code compiles and this new analysis works fine in the Solr admin analysis screen. But same fails with below exception when I try to index a document
> > > 30273 ERROR (qtp1689843956-18) [   x:gcom] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Exception writing document id id1 to the index; possible analysis error.
> > >         at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:180)
> > >         at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
> > >         at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
> > >         at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:934)
> > >         at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1089)
> > >         at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:712)
> > >         at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
> > > Caused by: java.lang.IllegalStateException: TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Java
> > > docs of TokenStream class for more information about the correct consuming workflow.
> > >         at org.apache.lucene.analysis.Tokenizer$1.read(Tokenizer.java:109)
> > >         at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:527)
> > >         at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:738)
> > >         at org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:159)
> > >         at com.synonyms.poc.synpoc.MySynonymFilterFactory.create(MySynonymFilterFactory.java:94)
> > >         at org.apache.solr.analysis.TokenizerChain.createComponents(TokenizerChain.java:91)
> > >         at org.apache.lucene.analysis.AnalyzerWrapper.createComponents(AnalyzerWrapper.java:101)
> > >         at org.apache.lucene.analysis.AnalyzerWrapper.createComponents(AnalyzerWrapper.java:101)
> > >         at org.apache.lucene.analysis.Analyzer.tokenStream(Analyzer.java:176)
> > >         at org.apache.lucene.document.Field.tokenStream(Field.java:562)
> > >         at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:628)
> > >         at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:365)
> > >         at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:321)
> > >         at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
> > >         at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
> > >         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
> > >         at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:282)
> > >         at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:214)
> > >         at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:169)
> > >         ... 37 more
> > >
> > > Any idea how can I read a token stream with out violating the token stream contract. I see a similar discussion here https://lucene.472066.n3.nabble.com/how-to-reuse-a-tokenStream-td850767.html, but doesn't help solve my problem.
> > >
> > > Also how come same error is not reported when analyzing the field value using Solr admin console analysis screen.
> > >
> > > Thanks
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]