exception to open a large index Insufficient system resources exist

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

exception to open a large index Insufficient system resources exist

Fang_Li
I met a problem to open an index bigger than 8GB and the following
exception was thrown. There is a segment which is bigger than 4GB
already. After searching internet, it is said that not using compound
index may solve the problem.

The same exception was thrown when merging with another index happens.
If the problem is caused by the big segment, we can adjust Lucene
parameters to control the segment size.
Anyone knows what's the cause for this exception? As reproducing the
problem will take a long time, your idea will be save us a lot of
effort.

java.io.IOException: Insufficient system resources exist to complete the
requested service
        at java.io.RandomAccessFile.readBytes(Native Method)
        at java.io.RandomAccessFile.read(RandomAccessFile.java:315)
        at
org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
y.java:596)
        at
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav
a:157)
        at
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j
ava:38)
        at
org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:78)
        at
org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.jav
a:73)
        at
org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
        at
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
        at
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:228)
        at
org.apache.lucene.index.MultiSegmentReader.<init>(MultiSegmentReader.jav
a:55)
        at
org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexRead
er.java:104)
        at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.j
ava:653)
        at
org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.j
ava:115)
        at
org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
        at
org.apache.lucene.index.IndexReader.open(IndexReader.java:227)

Thanks,
-Li


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: exception to open a large index Insufficient system resources exist

Danil ŢORIN
There should be no problem with large segments.
Please describe OS, FileSystem and JDK you are running on.

There might be some problems with file >2Gb on Win32/FAT, or in some
ancient Linuxes.

On Tue, Sep 1, 2009 at 12:37, <[hidden email]> wrote:

> I met a problem to open an index bigger than 8GB and the following
> exception was thrown. There is a segment which is bigger than 4GB
> already. After searching internet, it is said that not using compound
> index may solve the problem.
>
> The same exception was thrown when merging with another index happens.
> If the problem is caused by the big segment, we can adjust Lucene
> parameters to control the segment size.
> Anyone knows what's the cause for this exception? As reproducing the
> problem will take a long time, your idea will be save us a lot of
> effort.
>
> java.io.IOException: Insufficient system resources exist to complete the
> requested service
>        at java.io.RandomAccessFile.readBytes(Native Method)
>        at java.io.RandomAccessFile.read(RandomAccessFile.java:315)
>        at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
> y.java:596)
>        at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav
> a:157)
>        at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j
> ava:38)
>        at
> org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:78)
>        at
> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.jav
> a:73)
>        at
> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
>        at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
>        at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:228)
>        at
> org.apache.lucene.index.MultiSegmentReader.<init>(MultiSegmentReader.jav
> a:55)
>        at
> org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexRead
> er.java:104)
>        at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.j
> ava:653)
>        at
> org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.j
> ava:115)
>        at
> org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
>        at
> org.apache.lucene.index.IndexReader.open(IndexReader.java:227)
>
> Thanks,
> -Li
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: exception to open a large index Insufficient system resources exist

Fang_Li
We are running on Windows 2003 Enterprise Edition with NTFS file system on a local disc. JDK version is 1.5.0.12.

The problem was discussed before and there is no clear solution confirmed.

Thanks.

-----Original Message-----
From: Danil ŢORIN [mailto:[hidden email]]
Sent: Tuesday, September 01, 2009 5:56 PM
To: [hidden email]
Subject: Re: exception to open a large index Insufficient system resources exist

There should be no problem with large segments.
Please describe OS, FileSystem and JDK you are running on.

There might be some problems with file >2Gb on Win32/FAT, or in some
ancient Linuxes.

On Tue, Sep 1, 2009 at 12:37, <[hidden email]> wrote:

> I met a problem to open an index bigger than 8GB and the following
> exception was thrown. There is a segment which is bigger than 4GB
> already. After searching internet, it is said that not using compound
> index may solve the problem.
>
> The same exception was thrown when merging with another index happens.
> If the problem is caused by the big segment, we can adjust Lucene
> parameters to control the segment size.
> Anyone knows what's the cause for this exception? As reproducing the
> problem will take a long time, your idea will be save us a lot of
> effort.
>
> java.io.IOException: Insufficient system resources exist to complete the
> requested service
>        at java.io.RandomAccessFile.readBytes(Native Method)
>        at java.io.RandomAccessFile.read(RandomAccessFile.java:315)
>        at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
> y.java:596)
>        at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav
> a:157)
>        at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j
> ava:38)
>        at
> org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:78)
>        at
> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.jav
> a:73)
>        at
> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
>        at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
>        at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:228)
>        at
> org.apache.lucene.index.MultiSegmentReader.<init>(MultiSegmentReader.jav
> a:55)
>        at
> org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexRead
> er.java:104)
>        at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.j
> ava:653)
>        at
> org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.j
> ava:115)
>        at
> org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
>        at
> org.apache.lucene.index.IndexReader.open(IndexReader.java:227)
>
> Thanks,
> -Li
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

RE: exception to open a large index Insufficient system resources exist

Uwe Schindler
Which Lucene version, 64 bit JVM?

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [hidden email]

> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]]
> Sent: Tuesday, September 01, 2009 12:04 PM
> To: [hidden email]
> Subject: RE: exception to open a large index Insufficient system resources
> exist
>
> We are running on Windows 2003 Enterprise Edition with NTFS file system on
> a local disc. JDK version is 1.5.0.12.
>
> The problem was discussed before and there is no clear solution confirmed.
>
> Thanks.
>
> -----Original Message-----
> From: Danil ŢORIN [mailto:[hidden email]]
> Sent: Tuesday, September 01, 2009 5:56 PM
> To: [hidden email]
> Subject: Re: exception to open a large index Insufficient system resources
> exist
>
> There should be no problem with large segments.
> Please describe OS, FileSystem and JDK you are running on.
>
> There might be some problems with file >2Gb on Win32/FAT, or in some
> ancient Linuxes.
>
> On Tue, Sep 1, 2009 at 12:37, <[hidden email]> wrote:
> > I met a problem to open an index bigger than 8GB and the following
> > exception was thrown. There is a segment which is bigger than 4GB
> > already. After searching internet, it is said that not using compound
> > index may solve the problem.
> >
> > The same exception was thrown when merging with another index happens.
> > If the problem is caused by the big segment, we can adjust Lucene
> > parameters to control the segment size.
> > Anyone knows what's the cause for this exception? As reproducing the
> > problem will take a long time, your idea will be save us a lot of
> > effort.
> >
> > java.io.IOException: Insufficient system resources exist to complete the
> > requested service
> >        at java.io.RandomAccessFile.readBytes(Native Method)
> >        at java.io.RandomAccessFile.read(RandomAccessFile.java:315)
> >        at
> > org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
> > y.java:596)
> >        at
> > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav
> > a:157)
> >        at
> > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j
> > ava:38)
> >        at
> > org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:78)
> >        at
> > org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.jav
> > a:73)
> >        at
> > org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
> >        at
> > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
> >        at
> > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:228)
> >        at
> > org.apache.lucene.index.MultiSegmentReader.<init>(MultiSegmentReader.jav
> > a:55)
> >        at
> > org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexRead
> > er.java:104)
> >        at
> > org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.j
> > ava:653)
> >        at
> > org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.j
> > ava:115)
> >        at
> > org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
> >        at
> > org.apache.lucene.index.IndexReader.open(IndexReader.java:227)
> >
> > Thanks,
> > -Li
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: exception to open a large index Insufficient system resources exist

Fang_Li
32 bit JVM, 1.3G allocated heap size, Lucene 2.4.1.

In my option, this exception should not be caused by out of memory or out of system file handle because different exception should be thrown for these two cases.

Any hint?

Thanks,
Fang, Li

-----Original Message-----
From: Uwe Schindler [mailto:[hidden email]]
Sent: Tuesday, September 01, 2009 6:14 PM
To: [hidden email]
Subject: RE: exception to open a large index Insufficient system resources exist

Which Lucene version, 64 bit JVM?

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [hidden email]

> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]]
> Sent: Tuesday, September 01, 2009 12:04 PM
> To: [hidden email]
> Subject: RE: exception to open a large index Insufficient system resources
> exist
>
> We are running on Windows 2003 Enterprise Edition with NTFS file system on
> a local disc. JDK version is 1.5.0.12.
>
> The problem was discussed before and there is no clear solution confirmed.
>
> Thanks.
>
> -----Original Message-----
> From: Danil ŢORIN [mailto:[hidden email]]
> Sent: Tuesday, September 01, 2009 5:56 PM
> To: [hidden email]
> Subject: Re: exception to open a large index Insufficient system resources
> exist
>
> There should be no problem with large segments.
> Please describe OS, FileSystem and JDK you are running on.
>
> There might be some problems with file >2Gb on Win32/FAT, or in some
> ancient Linuxes.
>
> On Tue, Sep 1, 2009 at 12:37, <[hidden email]> wrote:
> > I met a problem to open an index bigger than 8GB and the following
> > exception was thrown. There is a segment which is bigger than 4GB
> > already. After searching internet, it is said that not using compound
> > index may solve the problem.
> >
> > The same exception was thrown when merging with another index happens.
> > If the problem is caused by the big segment, we can adjust Lucene
> > parameters to control the segment size.
> > Anyone knows what's the cause for this exception? As reproducing the
> > problem will take a long time, your idea will be save us a lot of
> > effort.
> >
> > java.io.IOException: Insufficient system resources exist to complete the
> > requested service
> >        at java.io.RandomAccessFile.readBytes(Native Method)
> >        at java.io.RandomAccessFile.read(RandomAccessFile.java:315)
> >        at
> > org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
> > y.java:596)
> >        at
> > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav
> > a:157)
> >        at
> > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j
> > ava:38)
> >        at
> > org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:78)
> >        at
> > org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.jav
> > a:73)
> >        at
> > org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
> >        at
> > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
> >        at
> > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:228)
> >        at
> > org.apache.lucene.index.MultiSegmentReader.<init>(MultiSegmentReader.jav
> > a:55)
> >        at
> > org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexRead
> > er.java:104)
> >        at
> > org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.j
> > ava:653)
> >        at
> > org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.j
> > ava:115)
> >        at
> > org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
> >        at
> > org.apache.lucene.index.IndexReader.open(IndexReader.java:227)
> >
> > Thanks,
> > -Li
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]