A flush exception in lucene 4.10.0

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

A flush exception in lucene 4.10.0

Yonghui Zhao
Hi all,

 Anyone see this exception before? Is it a lucene bug or something wrong in
my code?


Exception in thread "Thread-14" java.lang.IllegalArgumentException:
maxValue must be non-negative (got: -1)
        at
org.apache.lucene.util.packed.PackedInts.bitsRequired(PackedInts.java:1141)
        at
org.apache.lucene.codecs.lucene41.ForUtil.bitsRequired(ForUtil.java:253)
        at
org.apache.lucene.codecs.lucene41.ForUtil.writeBlock(ForUtil.java:174)
        at
org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.addPosition(Lucene41PostingsWriter.java:377)
        at
org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:486)
        at
org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:80)
        at
org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:114)
        at
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:441)
        at
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:510)
        at
org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:621)
        at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:414)
        at
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:292)
        at
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:267)
        at
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:257)
        at
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:171)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: A flush exception in lucene 4.10.0

Yonghui Zhao
Seems it is related to an empty segment, all docs in this segment are
deleted before commit.
Any one can confirm it? Maybe I need upgrade my lucene version.


2017-03-03 10:19 GMT+08:00 Yonghui Zhao <[hidden email]>:

> Hi all,
>
>  Anyone see this exception before? Is it a lucene bug or something wrong
> in my code?
>
>
> Exception in thread "Thread-14" java.lang.IllegalArgumentException:
> maxValue must be non-negative (got: -1)
>         at org.apache.lucene.util.packed.PackedInts.bitsRequired(
> PackedInts.java:1141)
>         at org.apache.lucene.codecs.lucene41.ForUtil.bitsRequired(
> ForUtil.java:253)
>         at org.apache.lucene.codecs.lucene41.ForUtil.writeBlock(
> ForUtil.java:174)
>         at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.
> addPosition(Lucene41PostingsWriter.java:377)
>         at org.apache.lucene.index.FreqProxTermsWriterPerField.flush(
> FreqProxTermsWriterPerField.java:486)
>         at org.apache.lucene.index.FreqProxTermsWriter.flush(
> FreqProxTermsWriter.java:80)
>         at org.apache.lucene.index.DefaultIndexingChain.flush(
> DefaultIndexingChain.java:114)
>         at org.apache.lucene.index.DocumentsWriterPerThread.flush(
> DocumentsWriterPerThread.java:441)
>         at org.apache.lucene.index.DocumentsWriter.doFlush(
> DocumentsWriter.java:510)
>         at org.apache.lucene.index.DocumentsWriter.flushAllThreads(
> DocumentsWriter.java:621)
>         at org.apache.lucene.index.IndexWriter.getReader(
> IndexWriter.java:414)
>         at org.apache.lucene.index.StandardDirectoryReader.
> doOpenFromWriter(StandardDirectoryReader.java:292)
>         at org.apache.lucene.index.StandardDirectoryReader.
> doOpenIfChanged(StandardDirectoryReader.java:267)
>         at org.apache.lucene.index.StandardDirectoryReader.
> doOpenIfChanged(StandardDirectoryReader.java:257)
>         at org.apache.lucene.index.DirectoryReader.openIfChanged(
> DirectoryReader.java:171)
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: A flush exception in lucene 4.10.0

Michael McCandless-2
This seems likely to be a Lucene bug, and it seems vaguely familiar.  I
tried to find the issue / commit that may have fixed it, but so far failed.

But 4.10.0 is truly ancient; you should at least try upgrading to 4.10.4?

Mike McCandless

http://blog.mikemccandless.com

On Wed, Mar 8, 2017 at 6:15 AM, Yonghui Zhao <[hidden email]> wrote:

> Seems it is related to an empty segment, all docs in this segment are
> deleted before commit.
> Any one can confirm it? Maybe I need upgrade my lucene version.
>
>
> 2017-03-03 10:19 GMT+08:00 Yonghui Zhao <[hidden email]>:
>
> > Hi all,
> >
> >  Anyone see this exception before? Is it a lucene bug or something wrong
> > in my code?
> >
> >
> > Exception in thread "Thread-14" java.lang.IllegalArgumentException:
> > maxValue must be non-negative (got: -1)
> >         at org.apache.lucene.util.packed.PackedInts.bitsRequired(
> > PackedInts.java:1141)
> >         at org.apache.lucene.codecs.lucene41.ForUtil.bitsRequired(
> > ForUtil.java:253)
> >         at org.apache.lucene.codecs.lucene41.ForUtil.writeBlock(
> > ForUtil.java:174)
> >         at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.
> > addPosition(Lucene41PostingsWriter.java:377)
> >         at org.apache.lucene.index.FreqProxTermsWriterPerField.flush(
> > FreqProxTermsWriterPerField.java:486)
> >         at org.apache.lucene.index.FreqProxTermsWriter.flush(
> > FreqProxTermsWriter.java:80)
> >         at org.apache.lucene.index.DefaultIndexingChain.flush(
> > DefaultIndexingChain.java:114)
> >         at org.apache.lucene.index.DocumentsWriterPerThread.flush(
> > DocumentsWriterPerThread.java:441)
> >         at org.apache.lucene.index.DocumentsWriter.doFlush(
> > DocumentsWriter.java:510)
> >         at org.apache.lucene.index.DocumentsWriter.flushAllThreads(
> > DocumentsWriter.java:621)
> >         at org.apache.lucene.index.IndexWriter.getReader(
> > IndexWriter.java:414)
> >         at org.apache.lucene.index.StandardDirectoryReader.
> > doOpenFromWriter(StandardDirectoryReader.java:292)
> >         at org.apache.lucene.index.StandardDirectoryReader.
> > doOpenIfChanged(StandardDirectoryReader.java:267)
> >         at org.apache.lucene.index.StandardDirectoryReader.
> > doOpenIfChanged(StandardDirectoryReader.java:257)
> >         at org.apache.lucene.index.DirectoryReader.openIfChanged(
> > DirectoryReader.java:171)
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: A flush exception in lucene 4.10.0

sarowe
Maybe <https://issues.apache.org/jira/browse/LUCENE-5116> (though it was committed in Lucene 4.5)?  Robert Muir pointed to this issue as fixing <https://issues.apache.org/jira/browse/SOLR-5144>, which contains a similar stack track to yours.

--
Steve
www.lucidworks.com

> On Mar 9, 2017, at 6:02 AM, Michael McCandless <[hidden email]> wrote:
>
> This seems likely to be a Lucene bug, and it seems vaguely familiar.  I
> tried to find the issue / commit that may have fixed it, but so far failed.
>
> But 4.10.0 is truly ancient; you should at least try upgrading to 4.10.4?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Wed, Mar 8, 2017 at 6:15 AM, Yonghui Zhao <[hidden email]> wrote:
>
>> Seems it is related to an empty segment, all docs in this segment are
>> deleted before commit.
>> Any one can confirm it? Maybe I need upgrade my lucene version.
>>
>>
>> 2017-03-03 10:19 GMT+08:00 Yonghui Zhao <[hidden email]>:
>>
>>> Hi all,
>>>
>>> Anyone see this exception before? Is it a lucene bug or something wrong
>>> in my code?
>>>
>>>
>>> Exception in thread "Thread-14" java.lang.IllegalArgumentException:
>>> maxValue must be non-negative (got: -1)
>>>        at org.apache.lucene.util.packed.PackedInts.bitsRequired(
>>> PackedInts.java:1141)
>>>        at org.apache.lucene.codecs.lucene41.ForUtil.bitsRequired(
>>> ForUtil.java:253)
>>>        at org.apache.lucene.codecs.lucene41.ForUtil.writeBlock(
>>> ForUtil.java:174)
>>>        at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.
>>> addPosition(Lucene41PostingsWriter.java:377)
>>>        at org.apache.lucene.index.FreqProxTermsWriterPerField.flush(
>>> FreqProxTermsWriterPerField.java:486)
>>>        at org.apache.lucene.index.FreqProxTermsWriter.flush(
>>> FreqProxTermsWriter.java:80)
>>>        at org.apache.lucene.index.DefaultIndexingChain.flush(
>>> DefaultIndexingChain.java:114)
>>>        at org.apache.lucene.index.DocumentsWriterPerThread.flush(
>>> DocumentsWriterPerThread.java:441)
>>>        at org.apache.lucene.index.DocumentsWriter.doFlush(
>>> DocumentsWriter.java:510)
>>>        at org.apache.lucene.index.DocumentsWriter.flushAllThreads(
>>> DocumentsWriter.java:621)
>>>        at org.apache.lucene.index.IndexWriter.getReader(
>>> IndexWriter.java:414)
>>>        at org.apache.lucene.index.StandardDirectoryReader.
>>> doOpenFromWriter(StandardDirectoryReader.java:292)
>>>        at org.apache.lucene.index.StandardDirectoryReader.
>>> doOpenIfChanged(StandardDirectoryReader.java:267)
>>>        at org.apache.lucene.index.StandardDirectoryReader.
>>> doOpenIfChanged(StandardDirectoryReader.java:257)
>>>        at org.apache.lucene.index.DirectoryReader.openIfChanged(
>>> DirectoryReader.java:171)
>>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: A flush exception in lucene 4.10.0

Yonghui Zhao
My version is 4.10.0 which is later than 4.5,  but I didn't find the fix
like
https://issues.apache.org/jira/secure/attachment/12593180/LUCENE-5116.patch  in
indexwriter, neither in 4.10.4. Not sure which version has fixed it.


public void addIndexes(IndexReader... readers) throws IOException {
    ensureOpen();
    int numDocs = 0;

    try {
      if (infoStream.isEnabled("IW")) {
        infoStream.message("IW", "flush at addIndexes(IndexReader...)");
      }
      flush(false, true);

      String mergedName = newSegmentName();
      final List<AtomicReader> mergeReaders = new ArrayList<>();
      for (IndexReader indexReader : readers) {
        numDocs += indexReader.numDocs();
        for (AtomicReaderContext ctx : indexReader.leaves()) {
          mergeReaders.add(ctx.reader());
        }
      }

      // Make sure adding the new documents to this index won't
      // exceed the limit:
      reserveDocs(numDocs);

      final IOContext context = new IOContext(new MergeInfo(numDocs, -1,
true, -1));

2017-03-09 22:44 GMT+08:00 Steve Rowe <[hidden email]>:

> Maybe <https://issues.apache.org/jira/browse/LUCENE-5116> (though it was
> committed in Lucene 4.5)?  Robert Muir pointed to this issue as fixing <
> https://issues.apache.org/jira/browse/SOLR-5144>, which contains a
> similar stack track to yours.
>
> --
> Steve
> www.lucidworks.com
>
> > On Mar 9, 2017, at 6:02 AM, Michael McCandless <
> [hidden email]> wrote:
> >
> > This seems likely to be a Lucene bug, and it seems vaguely familiar.  I
> > tried to find the issue / commit that may have fixed it, but so far
> failed.
> >
> > But 4.10.0 is truly ancient; you should at least try upgrading to 4.10.4?
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
> > On Wed, Mar 8, 2017 at 6:15 AM, Yonghui Zhao <[hidden email]>
> wrote:
> >
> >> Seems it is related to an empty segment, all docs in this segment are
> >> deleted before commit.
> >> Any one can confirm it? Maybe I need upgrade my lucene version.
> >>
> >>
> >> 2017-03-03 10:19 GMT+08:00 Yonghui Zhao <[hidden email]>:
> >>
> >>> Hi all,
> >>>
> >>> Anyone see this exception before? Is it a lucene bug or something wrong
> >>> in my code?
> >>>
> >>>
> >>> Exception in thread "Thread-14" java.lang.IllegalArgumentException:
> >>> maxValue must be non-negative (got: -1)
> >>>        at org.apache.lucene.util.packed.PackedInts.bitsRequired(
> >>> PackedInts.java:1141)
> >>>        at org.apache.lucene.codecs.lucene41.ForUtil.bitsRequired(
> >>> ForUtil.java:253)
> >>>        at org.apache.lucene.codecs.lucene41.ForUtil.writeBlock(
> >>> ForUtil.java:174)
> >>>        at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.
> >>> addPosition(Lucene41PostingsWriter.java:377)
> >>>        at org.apache.lucene.index.FreqProxTermsWriterPerField.flush(
> >>> FreqProxTermsWriterPerField.java:486)
> >>>        at org.apache.lucene.index.FreqProxTermsWriter.flush(
> >>> FreqProxTermsWriter.java:80)
> >>>        at org.apache.lucene.index.DefaultIndexingChain.flush(
> >>> DefaultIndexingChain.java:114)
> >>>        at org.apache.lucene.index.DocumentsWriterPerThread.flush(
> >>> DocumentsWriterPerThread.java:441)
> >>>        at org.apache.lucene.index.DocumentsWriter.doFlush(
> >>> DocumentsWriter.java:510)
> >>>        at org.apache.lucene.index.DocumentsWriter.flushAllThreads(
> >>> DocumentsWriter.java:621)
> >>>        at org.apache.lucene.index.IndexWriter.getReader(
> >>> IndexWriter.java:414)
> >>>        at org.apache.lucene.index.StandardDirectoryReader.
> >>> doOpenFromWriter(StandardDirectoryReader.java:292)
> >>>        at org.apache.lucene.index.StandardDirectoryReader.
> >>> doOpenIfChanged(StandardDirectoryReader.java:267)
> >>>        at org.apache.lucene.index.StandardDirectoryReader.
> >>> doOpenIfChanged(StandardDirectoryReader.java:257)
> >>>        at org.apache.lucene.index.DirectoryReader.openIfChanged(
> >>> DirectoryReader.java:171)
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: A flush exception in lucene 4.10.0

Yonghui Zhao
The exception in my case is related to one index field. The field is string
type, indexed, tokenized and stored, index option
is docsAndFreqsAndPositionsAndOffsets.

Each doc may index this field more than one times. For example doc 0 may
index it 3 times, 1st value is "A B C",  2nd is "D E F",3rd is "G H I".
If I concatenate these values with space, "A B C D E F G H I" , so each doc
only index this field one 1 time, but has the same effect, no exception
occurs.


2017-03-10 13:43 GMT+08:00 Yonghui Zhao <[hidden email]>:

> My version is 4.10.0 which is later than 4.5,  but I didn't find the fix
> like https://issues.apache.org/jira/secure/attachment/
> 12593180/LUCENE-5116.patch  in indexwriter, neither in 4.10.4. Not sure
> which version has fixed it.
>
>
> public void addIndexes(IndexReader... readers) throws IOException {
>     ensureOpen();
>     int numDocs = 0;
>
>     try {
>       if (infoStream.isEnabled("IW")) {
>         infoStream.message("IW", "flush at addIndexes(IndexReader...)");
>       }
>       flush(false, true);
>
>       String mergedName = newSegmentName();
>       final List<AtomicReader> mergeReaders = new ArrayList<>();
>       for (IndexReader indexReader : readers) {
>         numDocs += indexReader.numDocs();
>         for (AtomicReaderContext ctx : indexReader.leaves()) {
>           mergeReaders.add(ctx.reader());
>         }
>       }
>
>       // Make sure adding the new documents to this index won't
>       // exceed the limit:
>       reserveDocs(numDocs);
>
>       final IOContext context = new IOContext(new MergeInfo(numDocs, -1,
> true, -1));
>
> 2017-03-09 22:44 GMT+08:00 Steve Rowe <[hidden email]>:
>
>> Maybe <https://issues.apache.org/jira/browse/LUCENE-5116> (though it was
>> committed in Lucene 4.5)?  Robert Muir pointed to this issue as fixing <
>> https://issues.apache.org/jira/browse/SOLR-5144>, which contains a
>> similar stack track to yours.
>>
>> --
>> Steve
>> www.lucidworks.com
>>
>> > On Mar 9, 2017, at 6:02 AM, Michael McCandless <
>> [hidden email]> wrote:
>> >
>> > This seems likely to be a Lucene bug, and it seems vaguely familiar.  I
>> > tried to find the issue / commit that may have fixed it, but so far
>> failed.
>> >
>> > But 4.10.0 is truly ancient; you should at least try upgrading to
>> 4.10.4?
>> >
>> > Mike McCandless
>> >
>> > http://blog.mikemccandless.com
>> >
>> > On Wed, Mar 8, 2017 at 6:15 AM, Yonghui Zhao <[hidden email]>
>> wrote:
>> >
>> >> Seems it is related to an empty segment, all docs in this segment are
>> >> deleted before commit.
>> >> Any one can confirm it? Maybe I need upgrade my lucene version.
>> >>
>> >>
>> >> 2017-03-03 10:19 GMT+08:00 Yonghui Zhao <[hidden email]>:
>> >>
>> >>> Hi all,
>> >>>
>> >>> Anyone see this exception before? Is it a lucene bug or something
>> wrong
>> >>> in my code?
>> >>>
>> >>>
>> >>> Exception in thread "Thread-14" java.lang.IllegalArgumentException:
>> >>> maxValue must be non-negative (got: -1)
>> >>>        at org.apache.lucene.util.packed.PackedInts.bitsRequired(
>> >>> PackedInts.java:1141)
>> >>>        at org.apache.lucene.codecs.lucene41.ForUtil.bitsRequired(
>> >>> ForUtil.java:253)
>> >>>        at org.apache.lucene.codecs.lucene41.ForUtil.writeBlock(
>> >>> ForUtil.java:174)
>> >>>        at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.
>> >>> addPosition(Lucene41PostingsWriter.java:377)
>> >>>        at org.apache.lucene.index.FreqProxTermsWriterPerField.flush(
>> >>> FreqProxTermsWriterPerField.java:486)
>> >>>        at org.apache.lucene.index.FreqProxTermsWriter.flush(
>> >>> FreqProxTermsWriter.java:80)
>> >>>        at org.apache.lucene.index.DefaultIndexingChain.flush(
>> >>> DefaultIndexingChain.java:114)
>> >>>        at org.apache.lucene.index.DocumentsWriterPerThread.flush(
>> >>> DocumentsWriterPerThread.java:441)
>> >>>        at org.apache.lucene.index.DocumentsWriter.doFlush(
>> >>> DocumentsWriter.java:510)
>> >>>        at org.apache.lucene.index.DocumentsWriter.flushAllThreads(
>> >>> DocumentsWriter.java:621)
>> >>>        at org.apache.lucene.index.IndexWriter.getReader(
>> >>> IndexWriter.java:414)
>> >>>        at org.apache.lucene.index.StandardDirectoryReader.
>> >>> doOpenFromWriter(StandardDirectoryReader.java:292)
>> >>>        at org.apache.lucene.index.StandardDirectoryReader.
>> >>> doOpenIfChanged(StandardDirectoryReader.java:267)
>> >>>        at org.apache.lucene.index.StandardDirectoryReader.
>> >>> doOpenIfChanged(StandardDirectoryReader.java:257)
>> >>>        at org.apache.lucene.index.DirectoryReader.openIfChanged(
>> >>> DirectoryReader.java:171)
>> >>>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
Loading...