[jira] Created: (LUCENE-2501) ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (LUCENE-2501) ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice

JIRA jira@apache.org
ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice
----------------------------------------------------------

                 Key: LUCENE-2501
                 URL: https://issues.apache.org/jira/browse/LUCENE-2501
             Project: Lucene - Java
          Issue Type: Bug
          Components: Index
    Affects Versions: 3.0.1
            Reporter: Tim Smith


I'm seeing the following exception during indexing:
{code}
Caused by: java.lang.ArrayIndexOutOfBoundsException: 14
at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:118)
at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:490)
at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:511)
at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:104)
at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:120)
at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:468)
at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:174)
at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2085)
... 37 more
{code}


This seems to be caused by the following code:
{code}
    final int level = slice[upto] & 15;
    final int newLevel = nextLevelArray[level];
    final int newSize = levelSizeArray[newLevel];
{code}

this can result in "level" being a value between 0 and 14
the array nextLevelArray is only of size 10

i suspect the solution would be to either max the level to 10, or to add more entries to the nextLevelArray so it has 15 entries
however, i don't know if something more is going wrong here and this is just where the exception hits from a deeper issue



--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2501) ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879379#action_12879379 ]

Michael McCandless commented on LUCENE-2501:
--------------------------------------------

Hmmm, not good.  Can you boil this down to a smallish test case?

level should never be > 9, because nextLevelArray[*] is no greater than 9.  Something more serious is up...

> ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice
> ----------------------------------------------------------
>
>                 Key: LUCENE-2501
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2501
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.1
>            Reporter: Tim Smith
>
> I'm seeing the following exception during indexing:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 14
> at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:118)
> at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:490)
> at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:511)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:104)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:120)
> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:468)
> at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:174)
> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2085)
> ... 37 more
> {code}
> This seems to be caused by the following code:
> {code}
>     final int level = slice[upto] & 15;
>     final int newLevel = nextLevelArray[level];
>     final int newSize = levelSizeArray[newLevel];
> {code}
> this can result in "level" being a value between 0 and 14
> the array nextLevelArray is only of size 10
> i suspect the solution would be to either max the level to 10, or to add more entries to the nextLevelArray so it has 15 entries
> however, i don't know if something more is going wrong here and this is just where the exception hits from a deeper issue

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2501) ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879382#action_12879382 ]

Tim Smith commented on LUCENE-2501:
-----------------------------------

thats what i was afraid of

i got this report second hand, so i don't have access to the data that was being ingested

and i currently don't know enough about this section of the indexing code to guess in order to create a unit test
i'll try to create a test, but i expect it will be difficult (especially if no one else has ever seen this)

> ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice
> ----------------------------------------------------------
>
>                 Key: LUCENE-2501
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2501
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.1
>            Reporter: Tim Smith
>
> I'm seeing the following exception during indexing:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 14
> at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:118)
> at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:490)
> at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:511)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:104)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:120)
> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:468)
> at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:174)
> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2085)
> ... 37 more
> {code}
> This seems to be caused by the following code:
> {code}
>     final int level = slice[upto] & 15;
>     final int newLevel = nextLevelArray[level];
>     final int newSize = levelSizeArray[newLevel];
> {code}
> this can result in "level" being a value between 0 and 14
> the array nextLevelArray is only of size 10
> i suspect the solution would be to either max the level to 10, or to add more entries to the nextLevelArray so it has 15 entries
> however, i don't know if something more is going wrong here and this is just where the exception hits from a deeper issue

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2501) ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879389#action_12879389 ]

Michael McCandless commented on LUCENE-2501:
--------------------------------------------

Is this issue repeatable, on a different machine?

We do have a randomized test for this (TestByteSlices) -- I'll go start it w/ big random.multiplier.  Maybe it can uncover this ;)

> ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice
> ----------------------------------------------------------
>
>                 Key: LUCENE-2501
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2501
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.1
>            Reporter: Tim Smith
>
> I'm seeing the following exception during indexing:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 14
> at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:118)
> at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:490)
> at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:511)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:104)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:120)
> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:468)
> at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:174)
> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2085)
> ... 37 more
> {code}
> This seems to be caused by the following code:
> {code}
>     final int level = slice[upto] & 15;
>     final int newLevel = nextLevelArray[level];
>     final int newSize = levelSizeArray[newLevel];
> {code}
> this can result in "level" being a value between 0 and 14
> the array nextLevelArray is only of size 10
> i suspect the solution would be to either max the level to 10, or to add more entries to the nextLevelArray so it has 15 entries
> however, i don't know if something more is going wrong here and this is just where the exception hits from a deeper issue

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2501) ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879399#action_12879399 ]

Michael McCandless commented on LUCENE-2501:
--------------------------------------------

What sized RAM buffer was being used for IW when this exception happened?

> ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice
> ----------------------------------------------------------
>
>                 Key: LUCENE-2501
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2501
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.1
>            Reporter: Tim Smith
>
> I'm seeing the following exception during indexing:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 14
> at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:118)
> at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:490)
> at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:511)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:104)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:120)
> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:468)
> at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:174)
> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2085)
> ... 37 more
> {code}
> This seems to be caused by the following code:
> {code}
>     final int level = slice[upto] & 15;
>     final int newLevel = nextLevelArray[level];
>     final int newSize = levelSizeArray[newLevel];
> {code}
> this can result in "level" being a value between 0 and 14
> the array nextLevelArray is only of size 10
> i suspect the solution would be to either max the level to 10, or to add more entries to the nextLevelArray so it has 15 entries
> however, i don't know if something more is going wrong here and this is just where the exception hits from a deeper issue

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2501) ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879403#action_12879403 ]

Tim Smith commented on LUCENE-2501:
-----------------------------------

Here's all the info i have available right now (will try to get more):

16 core, 18-gig ram Windows 7 machine
1 JVM
16 index writers (each using default settings (64M ram, etc))
300+ docs/sec ingestion (small documents)
commit every 10 minutes
optimize every hour

The report i got indicated that every now and then one of these ArrayIndexOutOfBounds exceptions would occur
this would result in the document being indexed failing, but otherwise things would continue normally


> ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice
> ----------------------------------------------------------
>
>                 Key: LUCENE-2501
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2501
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.1
>            Reporter: Tim Smith
>
> I'm seeing the following exception during indexing:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 14
> at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:118)
> at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:490)
> at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:511)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:104)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:120)
> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:468)
> at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:174)
> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2085)
> ... 37 more
> {code}
> This seems to be caused by the following code:
> {code}
>     final int level = slice[upto] & 15;
>     final int newLevel = nextLevelArray[level];
>     final int newSize = levelSizeArray[newLevel];
> {code}
> this can result in "level" being a value between 0 and 14
> the array nextLevelArray is only of size 10
> i suspect the solution would be to either max the level to 10, or to add more entries to the nextLevelArray so it has 15 entries
> however, i don't know if something more is going wrong here and this is just where the exception hits from a deeper issue

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2501) ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879422#action_12879422 ]

Tim Smith commented on LUCENE-2501:
-----------------------------------

Some more info:

ingestion is being performed in multiple threads

ArrayIndexOutOfBounds exception is occurring in bursts
I suspect that these bursts of exceptions stop after the next commit (at which point the buffers are all reset)
NOTE: i have not yet confirmed this, but i suspect it


> ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice
> ----------------------------------------------------------
>
>                 Key: LUCENE-2501
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2501
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.1
>            Reporter: Tim Smith
>
> I'm seeing the following exception during indexing:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 14
> at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:118)
> at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:490)
> at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:511)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:104)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:120)
> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:468)
> at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:174)
> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2085)
> ... 37 more
> {code}
> This seems to be caused by the following code:
> {code}
>     final int level = slice[upto] & 15;
>     final int newLevel = nextLevelArray[level];
>     final int newSize = levelSizeArray[newLevel];
> {code}
> this can result in "level" being a value between 0 and 14
> the array nextLevelArray is only of size 10
> i suspect the solution would be to either max the level to 10, or to add more entries to the nextLevelArray so it has 15 entries
> however, i don't know if something more is going wrong here and this is just where the exception hits from a deeper issue

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2501) ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879483#action_12879483 ]

Tim Smith commented on LUCENE-2501:
-----------------------------------

Looks like this may be the original source of the errors

{code}
Caused by: org.apache.lucene.index.CorruptIndexException: docs out of order (607 <= 607 )
        at org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:76)
        at org.apache.lucene.index.FreqProxTermsWriter.appendPostings(FreqProxTermsWriter.java:209)
        at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:127)
        at org.apache.lucene.index.TermsHash.flush(TermsHash.java:144)
        at org.apache.lucene.index.DocInverter.flush(DocInverter.java:72)
        at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:64)
        at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:583)
        at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:3602)
        at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3511)
        at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3502)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2103)
{code}

> ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice
> ----------------------------------------------------------
>
>                 Key: LUCENE-2501
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2501
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.1
>            Reporter: Tim Smith
>
> I'm seeing the following exception during indexing:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 14
> at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:118)
> at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:490)
> at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:511)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:104)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:120)
> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:468)
> at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:174)
> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2085)
> ... 37 more
> {code}
> This seems to be caused by the following code:
> {code}
>     final int level = slice[upto] & 15;
>     final int newLevel = nextLevelArray[level];
>     final int newSize = levelSizeArray[newLevel];
> {code}
> this can result in "level" being a value between 0 and 14
> the array nextLevelArray is only of size 10
> i suspect the solution would be to either max the level to 10, or to add more entries to the nextLevelArray so it has 15 entries
> however, i don't know if something more is going wrong here and this is just where the exception hits from a deeper issue

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2501) ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879485#action_12879485 ]

Michael McCandless commented on LUCENE-2501:
--------------------------------------------

Are you certain about IW's RAM buffer size?  If the RAM buffer size was close to 2GB it could lead to exceptions like this.

> ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice
> ----------------------------------------------------------
>
>                 Key: LUCENE-2501
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2501
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.1
>            Reporter: Tim Smith
>
> I'm seeing the following exception during indexing:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 14
> at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:118)
> at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:490)
> at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:511)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:104)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:120)
> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:468)
> at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:174)
> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2085)
> ... 37 more
> {code}
> This seems to be caused by the following code:
> {code}
>     final int level = slice[upto] & 15;
>     final int newLevel = nextLevelArray[level];
>     final int newSize = levelSizeArray[newLevel];
> {code}
> this can result in "level" being a value between 0 and 14
> the array nextLevelArray is only of size 10
> i suspect the solution would be to either max the level to 10, or to add more entries to the nextLevelArray so it has 15 entries
> however, i don't know if something more is going wrong here and this is just where the exception hits from a deeper issue

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2501) ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879486#action_12879486 ]

Tim Smith commented on LUCENE-2501:
-----------------------------------

ram buffer size is set to 64.0

> ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice
> ----------------------------------------------------------
>
>                 Key: LUCENE-2501
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2501
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.1
>            Reporter: Tim Smith
>
> I'm seeing the following exception during indexing:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 14
> at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:118)
> at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:490)
> at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:511)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:104)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:120)
> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:468)
> at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:174)
> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2085)
> ... 37 more
> {code}
> This seems to be caused by the following code:
> {code}
>     final int level = slice[upto] & 15;
>     final int newLevel = nextLevelArray[level];
>     final int newSize = levelSizeArray[newLevel];
> {code}
> this can result in "level" being a value between 0 and 14
> the array nextLevelArray is only of size 10
> i suspect the solution would be to either max the level to 10, or to add more entries to the nextLevelArray so it has 15 entries
> however, i don't know if something more is going wrong here and this is just where the exception hits from a deeper issue

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2501) ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879487#action_12879487 ]

Michael McCandless commented on LUCENE-2501:
--------------------------------------------

Can you capture IW.setInfoStream output leading up to it?

> ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice
> ----------------------------------------------------------
>
>                 Key: LUCENE-2501
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2501
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.1
>            Reporter: Tim Smith
>
> I'm seeing the following exception during indexing:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 14
> at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:118)
> at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:490)
> at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:511)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:104)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:120)
> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:468)
> at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:174)
> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2085)
> ... 37 more
> {code}
> This seems to be caused by the following code:
> {code}
>     final int level = slice[upto] & 15;
>     final int newLevel = nextLevelArray[level];
>     final int newSize = levelSizeArray[newLevel];
> {code}
> this can result in "level" being a value between 0 and 14
> the array nextLevelArray is only of size 10
> i suspect the solution would be to either max the level to 10, or to add more entries to the nextLevelArray so it has 15 entries
> however, i don't know if something more is going wrong here and this is just where the exception hits from a deeper issue

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2501) ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879489#action_12879489 ]

Tim Smith commented on LUCENE-2501:
-----------------------------------

will do

may take some time before it occurs again

also, if this boils down to a synchronization error of some sort, the extra file io done to write the trace info to disk may add some implicit synchronization/slowdown that may result in not being able to reproduce the issue (i've seen this occur on non-lucene related synchronization issues, add the extra debug logging and it never fails anymore)

> ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice
> ----------------------------------------------------------
>
>                 Key: LUCENE-2501
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2501
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.1
>            Reporter: Tim Smith
>
> I'm seeing the following exception during indexing:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 14
> at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:118)
> at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:490)
> at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:511)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:104)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:120)
> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:468)
> at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:174)
> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2085)
> ... 37 more
> {code}
> This seems to be caused by the following code:
> {code}
>     final int level = slice[upto] & 15;
>     final int newLevel = nextLevelArray[level];
>     final int newSize = levelSizeArray[newLevel];
> {code}
> this can result in "level" being a value between 0 and 14
> the array nextLevelArray is only of size 10
> i suspect the solution would be to either max the level to 10, or to add more entries to the nextLevelArray so it has 15 entries
> however, i don't know if something more is going wrong here and this is just where the exception hits from a deeper issue

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2501) ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879495#action_12879495 ]

Michael McCandless commented on LUCENE-2501:
--------------------------------------------

bq. also, if this boils down to a synchronization error of some sort, the extra file io done to write the trace info to disk may add some implicit synchronization/slowdown that may result in not being able to reproduce the issue

Ahh yes, the Heisenbug (http://en.wikipedia.org/wiki/Unusual_software_bug#Heisenbug).  Still it's worth a shot to see if we can catch it in action...

> ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice
> ----------------------------------------------------------
>
>                 Key: LUCENE-2501
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2501
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.1
>            Reporter: Tim Smith
>
> I'm seeing the following exception during indexing:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 14
> at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:118)
> at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:490)
> at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:511)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:104)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:120)
> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:468)
> at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:174)
> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2085)
> ... 37 more
> {code}
> This seems to be caused by the following code:
> {code}
>     final int level = slice[upto] & 15;
>     final int newLevel = nextLevelArray[level];
>     final int newSize = levelSizeArray[newLevel];
> {code}
> this can result in "level" being a value between 0 and 14
> the array nextLevelArray is only of size 10
> i suspect the solution would be to either max the level to 10, or to add more entries to the nextLevelArray so it has 15 entries
> however, i don't know if something more is going wrong here and this is just where the exception hits from a deeper issue

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2501) ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881675#action_12881675 ]

Tim Smith commented on LUCENE-2501:
-----------------------------------

I've been informed that this exception is still happening

however, whenever index tracing is turned on, it never seems to occur (extra logging seems to be preventing some lower level synchronization issue from surfacing)






> ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice
> ----------------------------------------------------------
>
>                 Key: LUCENE-2501
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2501
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.1
>            Reporter: Tim Smith
>
> I'm seeing the following exception during indexing:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 14
> at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:118)
> at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:490)
> at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:511)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:104)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:120)
> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:468)
> at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:174)
> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2085)
> ... 37 more
> {code}
> This seems to be caused by the following code:
> {code}
>     final int level = slice[upto] & 15;
>     final int newLevel = nextLevelArray[level];
>     final int newSize = levelSizeArray[newLevel];
> {code}
> this can result in "level" being a value between 0 and 14
> the array nextLevelArray is only of size 10
> i suspect the solution would be to either max the level to 10, or to add more entries to the nextLevelArray so it has 15 entries
> however, i don't know if something more is going wrong here and this is just where the exception hits from a deeper issue

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2501) ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881923#action_12881923 ]

Michael McCandless commented on LUCENE-2501:
--------------------------------------------

Could you still post the infoStream output?  I can at least look at this to see how IW is being used.

Does this occur on different machines?  (Hardware issues could lead to exceptions like this).

> ArrayIndexOutOfBoundsException in ByteBlockPool.allocSlice
> ----------------------------------------------------------
>
>                 Key: LUCENE-2501
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2501
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.1
>            Reporter: Tim Smith
>
> I'm seeing the following exception during indexing:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 14
> at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:118)
> at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:490)
> at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:511)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:104)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:120)
> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:468)
> at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:174)
> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)
> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2085)
> ... 37 more
> {code}
> This seems to be caused by the following code:
> {code}
>     final int level = slice[upto] & 15;
>     final int newLevel = nextLevelArray[level];
>     final int newSize = levelSizeArray[newLevel];
> {code}
> this can result in "level" being a value between 0 and 14
> the array nextLevelArray is only of size 10
> i suspect the solution would be to either max the level to 10, or to add more entries to the nextLevelArray so it has 15 entries
> however, i don't know if something more is going wrong here and this is just where the exception hits from a deeper issue

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]