ArrayIndexOutOfBoundsException on BooleanScorer.score()

[Matt Magoffin]

Hello, I'm having a tough time trying to get to the root of an exception I
see sometimes on my Lucene 1.4.3 index. The exception is:

java.lang.ArrayIndexOutOfBoundsException: 4
 at org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:126)
 at org.apache.lucene.search.Scorer.score(Scorer.java:38)
 at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:92)
 at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:64)
 at org.apache.lucene.search.Hits.<init>(Hits.java:43)
 at org.apache.lucene.search.Searcher.search(Searcher.java:33)
 at org.apache.lucene.search.Searcher.search(Searcher.java:27)
 ...

and occurs when current.coord is set to 4 but coordFactors is of length 4.
 The maxCoord is value is 4 at this time, too. I can't get this to happen
every time I run the unit test that this error occurs in... which is very
frustrating, sorry.

I was wondering if anyone had an idea of what might cause this exception
(I'm assuming something is wrong on my use of Lucene, but I can't find
what).


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Matt Magoffin]

The exception does come on the heels of an update to the index by a
different thread than the one the search runs in. These log statements
show the operations going on just prior to the exception:

May-05 12:36:05 DEBUG - Indexing Lead 1024
May-05 12:36:06 TRACE - CON Closing IndexWriter
org.apache.lucene.index.IndexWriter@4b35ef
May-05 12:36:06 DEBUG - Parsing Lucene query string ...
May-05 12:36:06 DEBUG - Searching for leads with Lucene query: ...
May-05 12:36:06 TRACE - CON Thread[IndexQueueThread,5,main] releasing
WRITE lock for index [LEAD]
May-05 12:36:06 TRACE - CON Thread[main,5,main] got READ lock for index
[LEAD]
May-05 12:36:06 TRACE - CON Cached new IndexReader
org.apache.lucene.index.SegmentReader@15c80a4
May-05 12:36:06 TRACE - CON Created new IndexSearcher
org.apache.lucene.search.IndexSearcher@1c1f2
May-05 12:36:06 ERROR - Lucene exception during search on [LEAD]
java.lang.ArrayIndexOutOfBoundsException: 4
 ...
May-05 12:36:06 TRACE - CON Thread[main,5,main] releasing READ lock for
index [LEAD]

The search begins after the other thread's index update is complete,
however (as shown by the "releasing WRITE lock" and "got READ lock"
messages). Also a new IndexReader is created after the write completes in
this situation.

Any ideas are greatly appreciated.

-- m@

> Hello, I'm having a tough time trying to get to the root of an exception I
> see sometimes on my Lucene 1.4.3 index. The exception is:
>
> java.lang.ArrayIndexOutOfBoundsException: 4
>  at org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:126)
>  at org.apache.lucene.search.Scorer.score(Scorer.java:38)
>  at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:92)
>  at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:64)
>  at org.apache.lucene.search.Hits.<init>(Hits.java:43)
>  at org.apache.lucene.search.Searcher.search(Searcher.java:33)
>  at org.apache.lucene.search.Searcher.search(Searcher.java:27)
>  ...
>
> and occurs when current.coord is set to 4 but coordFactors is of length 4.
>  The maxCoord is value is 4 at this time, too. I can't get this to happen
> every time I run the unit test that this error occurs in... which is very
> frustrating, sorry.
>
> I was wondering if anyone had an idea of what might cause this exception
> (I'm assuming something is wrong on my use of Lucene, but I can't find
> what).
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Paul Elschot]

On Thursday 05 May 2005 21:45, Matt Magoffin wrote:

> The exception does come on the heels of an update to the index by a
> different thread than the one the search runs in. These log statements
> show the operations going on just prior to the exception:
>
> May-05 12:36:05 DEBUG - Indexing Lead 1024
> May-05 12:36:06 TRACE - CON Closing IndexWriter
> org.apache.lucene.index.IndexWriter@4b35ef
> May-05 12:36:06 DEBUG - Parsing Lucene query string ...
> May-05 12:36:06 DEBUG - Searching for leads with Lucene query: ...
> May-05 12:36:06 TRACE - CON Thread[IndexQueueThread,5,main] releasing
> WRITE lock for index [LEAD]
> May-05 12:36:06 TRACE - CON Thread[main,5,main] got READ lock for index
> [LEAD]
> May-05 12:36:06 TRACE - CON Cached new IndexReader
> org.apache.lucene.index.SegmentReader@15c80a4
> May-05 12:36:06 TRACE - CON Created new IndexSearcher
> org.apache.lucene.search.IndexSearcher@1c1f2
> May-05 12:36:06 ERROR - Lucene exception during search on [LEAD]
> java.lang.ArrayIndexOutOfBoundsException: 4
>  ...
> May-05 12:36:06 TRACE - CON Thread[main,5,main] releasing READ lock for
> index [LEAD]
>
> The search begins after the other thread's index update is complete,
> however (as shown by the "releasing WRITE lock" and "got READ lock"
> messages). Also a new IndexReader is created after the write completes in
> this situation.
>
> Any ideas are greatly appreciated.

The size of the coordFactors array depends on the number of non prohibited
clauses in the query. Could you indicate which query you are using?

And in case you find a way to reproduce this in a  test case, could
you file a bug report in bugzilla?

Regards,
Paul Elschot.

>
> -- m@
>
>
>
> > Hello, I'm having a tough time trying to get to the root of an exception I
> > see sometimes on my Lucene 1.4.3 index. The exception is:
> >
> > java.lang.ArrayIndexOutOfBoundsException: 4
> >  at org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:126)
> >  at org.apache.lucene.search.Scorer.score(Scorer.java:38)
> >  at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:92)
> >  at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:64)
> >  at org.apache.lucene.search.Hits.<init>(Hits.java:43)
> >  at org.apache.lucene.search.Searcher.search(Searcher.java:33)
> >  at org.apache.lucene.search.Searcher.search(Searcher.java:27)
> >  ...
> >
> > and occurs when current.coord is set to 4 but coordFactors is of length 4.
> >  The maxCoord is value is 4 at this time, too. I can't get this to happen
> > every time I run the unit test that this error occurs in... which is very
> > frustrating, sorry.
> >
> > I was wondering if anyone had an idea of what might cause this exception
> > (I'm assuming something is wrong on my use of Lucene, but I can't find
> > what).
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Matt Magoffin-2]

Here is a sample query that produces the error:

+dcode:123182 +cdate:[20050501 TO 20050531] +((+gn:m +sn:malen)
phone:[phone] addr:345 hiller (+sn:malen +zip:34535))

I'll see about getting a test case, but like I said it doesn't happen
every time so I've had a hard time tracking down the problem.

-- m@

> The size of the coordFactors array depends on the number of non prohibited
> clauses in the query. Could you indicate which query you are using?
>
> And in case you find a way to reproduce this in a  test case, could
> you file a bug report in bugzilla?
>
> Regards,
> Paul Elschot.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Paul Elschot]

On Friday 06 May 2005 18:19, Matt Magoffin wrote:
> Here is a sample query that produces the error:
>
> +dcode:123182 +cdate:[20050501 TO 20050531] +((+gn:m +sn:malen)
> phone:[phone] addr:345 hiller (+sn:malen +zip:34535))

From the earlier posted tracebacs, the problem is probably due to this
required clause:
((+gn:m +sn:malen) phone:[phone] addr:345 hiller (+sn:malen +zip:34535))
which has 4 optional subclauses, but the coordination factor array
does not seem to have an entry for 4 matching subclauses.

I shortly tried looking for possible causes in the source code (BooleanQuery,
BooleanScorer and QueryParser.jj), but I couldn't find any.

Regards,
Paul Elschot.

>
> I'll see about getting a test case, but like I said it doesn't happen
> every time so I've had a hard time tracking down the problem.
>
> -- m@
>
> > The size of the coordFactors array depends on the number of non prohibited
> > clauses in the query. Could you indicate which query you are using?
> >
> > And in case you find a way to reproduce this in a  test case, could
> > you file a bug report in bugzilla?
> >
> > Regards,
> > Paul Elschot.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Paul Elschot]

Matt,

On Thursday 05 May 2005 21:26, Matt Magoffin wrote:

> Hello, I'm having a tough time trying to get to the root of an exception I
> see sometimes on my Lucene 1.4.3 index. The exception is:
>
> java.lang.ArrayIndexOutOfBoundsException: 4
>  at org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:126)
>  at org.apache.lucene.search.Scorer.score(Scorer.java:38)
>  at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:92)
>  at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:64)
>  at org.apache.lucene.search.Hits.<init>(Hits.java:43)
>  at org.apache.lucene.search.Searcher.search(Searcher.java:33)
>  at org.apache.lucene.search.Searcher.search(Searcher.java:27)
>  ...
>
> and occurs when current.coord is set to 4 but coordFactors is of length 4.
>  The maxCoord is value is 4 at this time, too. I can't get this to happen
> every time I run the unit test that this error occurs in... which is very
> frustrating, sorry.
>
> I was wondering if anyone had an idea of what might cause this exception
> (I'm assuming something is wrong on my use of Lucene, but I can't find
> what).

It took some sleep to realize this:
This exception can happen when a scorer is add()'ed to the BooleanScorer after
the query search has begun. Given that it is difficult to reproduce, the odds
are that there two threads not properly synchronized: one add()ing
to the BooleanScorer and one searching the query too soon and
producing the above exception.

Regards,
Paul Elschot.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Matt Magoffin]

> It took some sleep to realize this:
> This exception can happen when a scorer is add()'ed to the BooleanScorer
> after
> the query search has begun. Given that it is difficult to reproduce, the
> odds
> are that there two threads not properly synchronized: one add()ing
> to the BooleanScorer and one searching the query too soon and
> producing the above exception.

Thanks for the tips. When you say "add()ing to the BooleanScorer" where do
you mean this is occurring? The query is constructed via QueryParser, and
I don't modify it in any way after it has been parsed.

-- m@


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]