Lucene 2.2.0 in 64-bit JVM: IndexReader is hung

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Lucene 2.2.0 in 64-bit JVM: IndexReader is hung

Venkat Rangan
Hi,

 

We have an application using Lucene 2.2.0 running in a Sun HotSpot JVM,
with JDK 1.6.0. We have had no problems with it in the 32-bit version of
the JVM. We recently upgraded to 64-bit JVM and occasionally, we are
observing a hang. In particular, the stack trace looks like this - the
RandomAccessFile.read() does not return ever - multiple stack traces are
hung on this, pointing to no bytes available to read for specific hits
returned by the searcher. We are aware of issue:
https://issues.apache.org/jira/browse/LUCENE-1262 which has a similar
stack trace. Is this a known problem or issue?

 

java.lang.Throwable

            at java.io.RandomAccessFile.readBytes(Native Method)

            at java.io.RandomAccessFile.read(RandomAccessFile.java:322)

            at
org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
y.java:550)

            at
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
java:122)

            at
org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(Com
poundFileReader.java:240)

            at
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav
a:143)

            at
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j
ava:38)

            at
org.apache.lucene.store.IndexInput.readChars(IndexInput.java:107)

            at
org.apache.lucene.store.IndexInput.readString(IndexInput.java:93)

            at
org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:253)

            at
org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:121)

            at
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:346)

            at
org.apache.lucene.index.MultiReader.document(MultiReader.java:117)

            at
org.apache.lucene.index.MultiReader.document(MultiReader.java:106)

            at
org.apache.lucene.index.IndexReader.document(IndexReader.java:437)

 

The call is something like:

 

              try {

                  IndexReader indexReader = getLuIndexReader();

                  if (indexReader.isDeleted(docId) || docId >=
indexReader.maxDoc())

                        return null;

                  lu_Document = lu_indexReader.document(docId);

                  return lu_Document;

              } catch (IOException e) {

                  ...

  }

Thanks,

 

-venkat

 

Reply | Threaded
Open this post in threaded view
|

Re: Lucene 2.2.0 in 64-bit JVM: IndexReader is hung

Yonik Seeley-2-2
So even when you stop searches, you observe a thread stuck on
RandomAccessFile.readBytes()?  In an active system, it would be normal
to see threads blocked there... just on different calls.

If RandomAccessFile.readBytes() is actually hanging, it's not a Lucene
issue, but a JVM/OS bug.


-Yonik
http://www.lucidimagination.com


On Mon, Mar 23, 2009 at 3:31 PM, Venkat Rangan
<[hidden email]> wrote:

> Hi,
>
>
>
> We have an application using Lucene 2.2.0 running in a Sun HotSpot JVM,
> with JDK 1.6.0. We have had no problems with it in the 32-bit version of
> the JVM. We recently upgraded to 64-bit JVM and occasionally, we are
> observing a hang. In particular, the stack trace looks like this - the
> RandomAccessFile.read() does not return ever - multiple stack traces are
> hung on this, pointing to no bytes available to read for specific hits
> returned by the searcher. We are aware of issue:
> https://issues.apache.org/jira/browse/LUCENE-1262 which has a similar
> stack trace. Is this a known problem or issue?
>
>
>
> java.lang.Throwable
>
>            at java.io.RandomAccessFile.readBytes(Native Method)
>
>            at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
>
>            at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
> y.java:550)
>
>            at
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
> java:122)
>
>            at
> org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(Com
> poundFileReader.java:240)
>
>            at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav
> a:143)
>
>            at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j
> ava:38)
>
>            at
> org.apache.lucene.store.IndexInput.readChars(IndexInput.java:107)
>
>            at
> org.apache.lucene.store.IndexInput.readString(IndexInput.java:93)
>
>            at
> org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:253)
>
>            at
> org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:121)
>
>            at
> org.apache.lucene.index.SegmentReader.document(SegmentReader.java:346)
>
>            at
> org.apache.lucene.index.MultiReader.document(MultiReader.java:117)
>
>            at
> org.apache.lucene.index.MultiReader.document(MultiReader.java:106)
>
>            at
> org.apache.lucene.index.IndexReader.document(IndexReader.java:437)
>
>
>
> The call is something like:
>
>
>
>              try {
>
>                  IndexReader indexReader = getLuIndexReader();
>
>                  if (indexReader.isDeleted(docId) || docId >=
> indexReader.maxDoc())
>
>                        return null;
>
>                  lu_Document = lu_indexReader.document(docId);
>
>                  return lu_Document;
>
>              } catch (IOException e) {
>
>                  ...
>
>  }
>
> Thanks,
>
>
>
> -venkat

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Lucene 2.2.0 in 64-bit JVM: IndexReader is hung

Venkat Rangan
Yonik,

Thanks for your response. It is actually hanging on RandomAccessFile.readBytes() when there are in fact bytes to read. Switching to 32-bit JVM does not hang against the same index. Also, as you point out, this may be a JVM/OS issue, but Lucene just exposes it.

-venkat

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Yonik Seeley
Sent: Monday, March 23, 2009 1:06 PM
To: [hidden email]
Subject: Re: Lucene 2.2.0 in 64-bit JVM: IndexReader is hung

So even when you stop searches, you observe a thread stuck on
RandomAccessFile.readBytes()?  In an active system, it would be normal
to see threads blocked there... just on different calls.

If RandomAccessFile.readBytes() is actually hanging, it's not a Lucene
issue, but a JVM/OS bug.


-Yonik
http://www.lucidimagination.com


On Mon, Mar 23, 2009 at 3:31 PM, Venkat Rangan
<[hidden email]> wrote:

> Hi,
>
>
>
> We have an application using Lucene 2.2.0 running in a Sun HotSpot JVM,
> with JDK 1.6.0. We have had no problems with it in the 32-bit version of
> the JVM. We recently upgraded to 64-bit JVM and occasionally, we are
> observing a hang. In particular, the stack trace looks like this - the
> RandomAccessFile.read() does not return ever - multiple stack traces are
> hung on this, pointing to no bytes available to read for specific hits
> returned by the searcher. We are aware of issue:
> https://issues.apache.org/jira/browse/LUCENE-1262 which has a similar
> stack trace. Is this a known problem or issue?
>
>
>
> java.lang.Throwable
>
>            at java.io.RandomAccessFile.readBytes(Native Method)
>
>            at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
>
>            at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
> y.java:550)
>
>            at
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
> java:122)
>
>            at
> org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(Com
> poundFileReader.java:240)
>
>            at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav
> a:143)
>
>            at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j
> ava:38)
>
>            at
> org.apache.lucene.store.IndexInput.readChars(IndexInput.java:107)
>
>            at
> org.apache.lucene.store.IndexInput.readString(IndexInput.java:93)
>
>            at
> org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:253)
>
>            at
> org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:121)
>
>            at
> org.apache.lucene.index.SegmentReader.document(SegmentReader.java:346)
>
>            at
> org.apache.lucene.index.MultiReader.document(MultiReader.java:117)
>
>            at
> org.apache.lucene.index.MultiReader.document(MultiReader.java:106)
>
>            at
> org.apache.lucene.index.IndexReader.document(IndexReader.java:437)
>
>
>
> The call is something like:
>
>
>
>              try {
>
>                  IndexReader indexReader = getLuIndexReader();
>
>                  if (indexReader.isDeleted(docId) || docId >=
> indexReader.maxDoc())
>
>                        return null;
>
>                  lu_Document = lu_indexReader.document(docId);
>
>                  return lu_Document;
>
>              } catch (IOException e) {
>
>                  ...
>
>  }
>
> Thanks,
>
>
>
> -venkat

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene 2.2.0 in 64-bit JVM: IndexReader is hung

Michael McCandless-2

Still, it's good to raise such issues here, and track them with Jira.
Even if it's not a Lucene issue, it'd be great to help Sun fix the JRE
issue.  It's possible to reduce the Lucene case down to a simpler test
case to give to Sun.  This is what happened with LUCENE-1282, which
resulted in Sun finally fixing the JRE bug.

There are various other "must be JRE/OS bug" issues...eg LUCENE-1342
(segv in older versions of Lucene on a 64 bit JRE) which is still open
and LUCENE-1566 (getting a false OOME on a large index while reading
norms).

Your situation sounds different from all of these... so could you
please open a Jira issue and post as many details as possible,
including a full stack trace of all threads (not just the hung ones)?

I think the similarity of the stack trace in LUCENE-1262 is
coincidental: it's just that you're also reading stored fields when
the hang hits.

Mike

Venkat Rangan wrote:

> Yonik,
>
> Thanks for your response. It is actually hanging on  
> RandomAccessFile.readBytes() when there are in fact bytes to read.  
> Switching to 32-bit JVM does not hang against the same index. Also,  
> as you point out, this may be a JVM/OS issue, but Lucene just  
> exposes it.
>
> -venkat
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of  
> Yonik Seeley
> Sent: Monday, March 23, 2009 1:06 PM
> To: [hidden email]
> Subject: Re: Lucene 2.2.0 in 64-bit JVM: IndexReader is hung
>
> So even when you stop searches, you observe a thread stuck on
> RandomAccessFile.readBytes()?  In an active system, it would be normal
> to see threads blocked there... just on different calls.
>
> If RandomAccessFile.readBytes() is actually hanging, it's not a Lucene
> issue, but a JVM/OS bug.
>
>
> -Yonik
> http://www.lucidimagination.com
>
>
> On Mon, Mar 23, 2009 at 3:31 PM, Venkat Rangan
> <[hidden email]> wrote:
>> Hi,
>>
>>
>>
>> We have an application using Lucene 2.2.0 running in a Sun HotSpot  
>> JVM,
>> with JDK 1.6.0. We have had no problems with it in the 32-bit  
>> version of
>> the JVM. We recently upgraded to 64-bit JVM and occasionally, we are
>> observing a hang. In particular, the stack trace looks like this -  
>> the
>> RandomAccessFile.read() does not return ever - multiple stack  
>> traces are
>> hung on this, pointing to no bytes available to read for specific  
>> hits
>> returned by the searcher. We are aware of issue:
>> https://issues.apache.org/jira/browse/LUCENE-1262 which has a similar
>> stack trace. Is this a known problem or issue?
>>
>>
>>
>> java.lang.Throwable
>>
>>            at java.io.RandomAccessFile.readBytes(Native Method)
>>
>>            at java.io.RandomAccessFile.read(RandomAccessFile.java:
>> 322)
>>
>>            at
>> org.apache.lucene.store.FSDirectory
>> $FSIndexInput.readInternal(FSDirector
>> y.java:550)
>>
>>            at
>> org
>> .apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
>> java:122)
>>
>>            at
>> org.apache.lucene.index.CompoundFileReader
>> $CSIndexInput.readInternal(Com
>> poundFileReader.java:240)
>>
>>            at
>> org
>> .apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav
>> a:143)
>>
>>            at
>> org
>> .apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j
>> ava:38)
>>
>>            at
>> org.apache.lucene.store.IndexInput.readChars(IndexInput.java:107)
>>
>>            at
>> org.apache.lucene.store.IndexInput.readString(IndexInput.java:93)
>>
>>            at
>> org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:253)
>>
>>            at
>> org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:121)
>>
>>            at
>> org.apache.lucene.index.SegmentReader.document(SegmentReader.java:
>> 346)
>>
>>            at
>> org.apache.lucene.index.MultiReader.document(MultiReader.java:117)
>>
>>            at
>> org.apache.lucene.index.MultiReader.document(MultiReader.java:106)
>>
>>            at
>> org.apache.lucene.index.IndexReader.document(IndexReader.java:437)
>>
>>
>>
>> The call is something like:
>>
>>
>>
>>              try {
>>
>>                  IndexReader indexReader = getLuIndexReader();
>>
>>                  if (indexReader.isDeleted(docId) || docId >=
>> indexReader.maxDoc())
>>
>>                        return null;
>>
>>                  lu_Document = lu_indexReader.document(docId);
>>
>>                  return lu_Document;
>>
>>              } catch (IOException e) {
>>
>>                  ...
>>
>>  }
>>
>> Thanks,
>>
>>
>>
>> -venkat
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]