“Too many open files" error hit when I run my Lucene 3.0.3 application on Java 8

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

“Too many open files" error hit when I run my Lucene 3.0.3 application on Java 8

Leonid Bolshinsky
I have a search engine based on Lucene 3.0.3 and I can't change the Lucene
version for reasons that are out of scope of this question. Now I have a
requirement to move from Java 6 to Java 8, however when I run the indexing
using Java 8 JVM, I hit "Too many open files issue" as below:

java.io.FileNotFoundException: /myIndex/_27c.fdx (Too many open files)
    at java.io.RandomAccessFile.open0(Native Method)
    at java.io.RandomAccessFile.open(RandomAccessFile.java:333)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:257)
    at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:180)
    at org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:85)
    at org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:86)
    at org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66)
    at org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144)
    at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:192)
    at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1559)
    at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1578)
    at org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1155)
    at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:832)
    at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:807)
    at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2155)

The ulimit on my system is 80.000 files and when I run lsof, I get that the
open files belong to the index and they are marked as (deleted) segments. I
am aware that there are tons of questions, answers and blogs about the
Lucene open files issue, however:

   - I've already tried whatever is recommended (review the code looking
   for open IndexWriters, increase ulimit, enable compound mode, decrease the
   number of segments from 10 to 4, review the code for open files) without
   success and
   - The issue didn't appear with Java 6 so I am guessing that the problem
   is in how Java 8 handles files.

Anybody faced this issue with Java 8 before? Any additional idea about how
to further troubleshoot this issue or what could be the cause?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: “Too many open files" error hit when I run my Lucene 3.0.3 application on Java 8

Frederik Van Hoyweghen
Did you make any changes to your code between running on java 6 and 8?
The java file API has changed considerably since java 8.

If you -have- rewritten some of the file handling code in your indexing
process, make sure to explicitly close the streams you create, or use
the (since java 7) try-with-resources construct.

On 22/02/2017 16:18, Leonid Bolshinsky wrote:

> I have a search engine based on Lucene 3.0.3 and I can't change the Lucene
> version for reasons that are out of scope of this question. Now I have a
> requirement to move from Java 6 to Java 8, however when I run the indexing
> using Java 8 JVM, I hit "Too many open files issue" as below:
>
> java.io.FileNotFoundException: /myIndex/_27c.fdx (Too many open files)
>      at java.io.RandomAccessFile.open0(Native Method)
>      at java.io.RandomAccessFile.open(RandomAccessFile.java:333)
>      at java.io.RandomAccessFile.<init>(RandomAccessFile.java:257)
>      at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:180)
>      at org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:85)
>      at org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:86)
>      at org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66)
>      at org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144)
>      at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:192)
>      at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1559)
>      at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1578)
>      at org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1155)
>      at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:832)
>      at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:807)
>      at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2155)
>
> The ulimit on my system is 80.000 files and when I run lsof, I get that the
> open files belong to the index and they are marked as (deleted) segments. I
> am aware that there are tons of questions, answers and blogs about the
> Lucene open files issue, however:
>
>     - I've already tried whatever is recommended (review the code looking
>     for open IndexWriters, increase ulimit, enable compound mode, decrease the
>     number of segments from 10 to 4, review the code for open files) without
>     success and
>     - The issue didn't appear with Java 6 so I am guessing that the problem
>     is in how Java 8 handles files.
>
> Anybody faced this issue with Java 8 before? Any additional idea about how
> to further troubleshoot this issue or what could be the cause?
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: “Too many open files" error hit when I run my Lucene 3.0.3 application on Java 8

Leonid Bolshinsky
No code changes between running in Java 6 and Java 8. The same code still
runs successfully with Java 6 while is failing with Java 8.

On Feb 22, 2017 5:52 PM, "Frederik Van Hoyweghen" <
[hidden email]> wrote:

Did you make any changes to your code between running on java 6 and 8?
The java file API has changed considerably since java 8.

If you -have- rewritten some of the file handling code in your indexing
process, make sure to explicitly close the streams you create, or use the
(since java 7) try-with-resources construct.


On 22/02/2017 16:18, Leonid Bolshinsky wrote:

> I have a search engine based on Lucene 3.0.3 and I can't change the Lucene
> version for reasons that are out of scope of this question. Now I have a
> requirement to move from Java 6 to Java 8, however when I run the indexing
> using Java 8 JVM, I hit "Too many open files issue" as below:
>
> java.io.FileNotFoundException: /myIndex/_27c.fdx (Too many open files)
>      at java.io.RandomAccessFile.open0(Native Method)
>      at java.io.RandomAccessFile.open(RandomAccessFile.java:333)
>      at java.io.RandomAccessFile.<init>(RandomAccessFile.java:257)
>      at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutpu
> t.<init>(SimpleFSDirectory.java:180)
>      at org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDir
> ectory.java:85)
>      at org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:86)
>      at org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(
> StoredFieldsWriter.java:66)
>      at org.apache.lucene.index.StoredFieldsWriter.finishDocument(St
> oredFieldsWriter.java:144)
>      at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(
> StoredFieldsWriter.java:192)
>      at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocum
> ent(DocumentsWriter.java:1559)
>      at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(Docume
> ntsWriter.java:1578)
>      at org.apache.lucene.index.DocumentsWriter.finishDocument(Docum
> entsWriter.java:1155)
>      at org.apache.lucene.index.DocumentsWriter.updateDocument(Docum
> entsWriter.java:832)
>      at org.apache.lucene.index.DocumentsWriter.updateDocument(Docum
> entsWriter.java:807)
>      at org.apache.lucene.index.IndexWriter.updateDocument(IndexWrit
> er.java:2155)
>
> The ulimit on my system is 80.000 files and when I run lsof, I get that the
> open files belong to the index and they are marked as (deleted) segments. I
> am aware that there are tons of questions, answers and blogs about the
> Lucene open files issue, however:
>
>     - I've already tried whatever is recommended (review the code looking
>
>     for open IndexWriters, increase ulimit, enable compound mode, decrease
> the
>     number of segments from 10 to 4, review the code for open files)
> without
>     success and
>     - The issue didn't appear with Java 6 so I am guessing that the problem
>
>     is in how Java 8 handles files.
>
> Anybody faced this issue with Java 8 before? Any additional idea about how
> to further troubleshoot this issue or what could be the cause?
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: “Too many open files" error hit when I run my Lucene 3.0.3 application on Java 8

Torsten Krah-2
What does lsof tell - which handles are open and which are unexpected or
suspicious?

Am 22.02.2017 5:27 nachm. schrieb "Leonid Bolshinsky" <[hidden email]>:

> No code changes between running in Java 6 and Java 8. The same code still
> runs successfully with Java 6 while is failing with Java 8.
>
> On Feb 22, 2017 5:52 PM, "Frederik Van Hoyweghen" <
> [hidden email]> wrote:
>
> Did you make any changes to your code between running on java 6 and 8?
> The java file API has changed considerably since java 8.
>
> If you -have- rewritten some of the file handling code in your indexing
> process, make sure to explicitly close the streams you create, or use the
> (since java 7) try-with-resources construct.
>
>
> On 22/02/2017 16:18, Leonid Bolshinsky wrote:
>
> > I have a search engine based on Lucene 3.0.3 and I can't change the
> Lucene
> > version for reasons that are out of scope of this question. Now I have a
> > requirement to move from Java 6 to Java 8, however when I run the
> indexing
> > using Java 8 JVM, I hit "Too many open files issue" as below:
> >
> > java.io.FileNotFoundException: /myIndex/_27c.fdx (Too many open files)
> >      at java.io.RandomAccessFile.open0(Native Method)
> >      at java.io.RandomAccessFile.open(RandomAccessFile.java:333)
> >      at java.io.RandomAccessFile.<init>(RandomAccessFile.java:257)
> >      at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutpu
> > t.<init>(SimpleFSDirectory.java:180)
> >      at org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDir
> > ectory.java:85)
> >      at org.apache.lucene.index.FieldsWriter.<init>(
> FieldsWriter.java:86)
> >      at org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(
> > StoredFieldsWriter.java:66)
> >      at org.apache.lucene.index.StoredFieldsWriter.finishDocument(St
> > oredFieldsWriter.java:144)
> >      at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(
> > StoredFieldsWriter.java:192)
> >      at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocum
> > ent(DocumentsWriter.java:1559)
> >      at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(Docume
> > ntsWriter.java:1578)
> >      at org.apache.lucene.index.DocumentsWriter.finishDocument(Docum
> > entsWriter.java:1155)
> >      at org.apache.lucene.index.DocumentsWriter.updateDocument(Docum
> > entsWriter.java:832)
> >      at org.apache.lucene.index.DocumentsWriter.updateDocument(Docum
> > entsWriter.java:807)
> >      at org.apache.lucene.index.IndexWriter.updateDocument(IndexWrit
> > er.java:2155)
> >
> > The ulimit on my system is 80.000 files and when I run lsof, I get that
> the
> > open files belong to the index and they are marked as (deleted)
> segments. I
> > am aware that there are tons of questions, answers and blogs about the
> > Lucene open files issue, however:
> >
> >     - I've already tried whatever is recommended (review the code looking
> >
> >     for open IndexWriters, increase ulimit, enable compound mode,
> decrease
> > the
> >     number of segments from 10 to 4, review the code for open files)
> > without
> >     success and
> >     - The issue didn't appear with Java 6 so I am guessing that the
> problem
> >
> >     is in how Java 8 handles files.
> >
> > Anybody faced this issue with Java 8 before? Any additional idea about
> how
> > to further troubleshoot this issue or what could be the cause?
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: “Too many open files" error hit when I run my Lucene 3.0.3 application on Java 8

Uwe Schindler
In reply to this post by Leonid Bolshinsky
The garbage collector in Java 8 may have different behavior. This may cause issues if you have buggy code that does not close indexreaders open in parallel to indexing. If it relies on Garbage Collector to close accidentally open files, that won't work anymore on Java 8. The reason for this might be changes to the finalists of Random access files in this issue for Java 7: https://bugs.openjdk.java.net/browse/JDK-6322678

Please make sure that you close all IndexWriters and IndexReaders after use, so cleanup does not rely on GC.

I am also not sure if there are bugs in this old Lucene version that might fail to close all streams correctly. This may cause these issues in combination with the above fixes in Java 8.

Uwe

Am 22. Februar 2017 17:27:48 MEZ schrieb Leonid Bolshinsky <[hidden email]>:

>No code changes between running in Java 6 and Java 8. The same code
>still
>runs successfully with Java 6 while is failing with Java 8.
>
>On Feb 22, 2017 5:52 PM, "Frederik Van Hoyweghen" <
>[hidden email]> wrote:
>
>Did you make any changes to your code between running on java 6 and 8?
>The java file API has changed considerably since java 8.
>
>If you -have- rewritten some of the file handling code in your indexing
>process, make sure to explicitly close the streams you create, or use
>the
>(since java 7) try-with-resources construct.
>
>
>On 22/02/2017 16:18, Leonid Bolshinsky wrote:
>
>> I have a search engine based on Lucene 3.0.3 and I can't change the
>Lucene
>> version for reasons that are out of scope of this question. Now I
>have a
>> requirement to move from Java 6 to Java 8, however when I run the
>indexing
>> using Java 8 JVM, I hit "Too many open files issue" as below:
>>
>> java.io.FileNotFoundException: /myIndex/_27c.fdx (Too many open
>files)
>>      at java.io.RandomAccessFile.open0(Native Method)
>>      at java.io.RandomAccessFile.open(RandomAccessFile.java:333)
>>      at java.io.RandomAccessFile.<init>(RandomAccessFile.java:257)
>>      at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutpu
>> t.<init>(SimpleFSDirectory.java:180)
>>      at org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDir
>> ectory.java:85)
>>      at
>org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:86)
>>      at org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(
>> StoredFieldsWriter.java:66)
>>      at org.apache.lucene.index.StoredFieldsWriter.finishDocument(St
>> oredFieldsWriter.java:144)
>>      at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(
>> StoredFieldsWriter.java:192)
>>      at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocum
>> ent(DocumentsWriter.java:1559)
>>      at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(Docume
>> ntsWriter.java:1578)
>>      at org.apache.lucene.index.DocumentsWriter.finishDocument(Docum
>> entsWriter.java:1155)
>>      at org.apache.lucene.index.DocumentsWriter.updateDocument(Docum
>> entsWriter.java:832)
>>      at org.apache.lucene.index.DocumentsWriter.updateDocument(Docum
>> entsWriter.java:807)
>>      at org.apache.lucene.index.IndexWriter.updateDocument(IndexWrit
>> er.java:2155)
>>
>> The ulimit on my system is 80.000 files and when I run lsof, I get
>that the
>> open files belong to the index and they are marked as (deleted)
>segments. I
>> am aware that there are tons of questions, answers and blogs about
>the
>> Lucene open files issue, however:
>>
>>     - I've already tried whatever is recommended (review the code
>looking
>>
>>     for open IndexWriters, increase ulimit, enable compound mode,
>decrease
>> the
>>     number of segments from 10 to 4, review the code for open files)
>> without
>>     success and
>>     - The issue didn't appear with Java 6 so I am guessing that the
>problem
>>
>>     is in how Java 8 handles files.
>>
>> Anybody faced this issue with Java 8 before? Any additional idea
>about how
>> to further troubleshoot this issue or what could be the cause?
>>
>>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: [hidden email]
>For additional commands, e-mail: [hidden email]

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
Loading...