[jira] Created: (LUCENE-1192) FileNotFound exception on adding a RAMDirectory to an IndexWriter

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (LUCENE-1192) FileNotFound exception on adding a RAMDirectory to an IndexWriter

ASF GitHub Bot (Jira)
FileNotFound exception on adding a RAMDirectory to an IndexWriter
-----------------------------------------------------------------

                 Key: LUCENE-1192
                 URL: https://issues.apache.org/jira/browse/LUCENE-1192
             Project: Lucene - Java
          Issue Type: Bug
    Affects Versions: 2.3.1
            Reporter: Alex Falca


Hi,

I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
{noformat}
java.io.FileNotFoundException: _8.fnm
        at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
        at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
        at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
        at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
        at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
{noformat}

Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
1. First documents are added to and index which use a RAMDirectory as a storage
2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.

Sample code:
{code}
IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
mainIndexWriter.setMergeFactor(10);
mainIndexWriter.setUseCompoundFile(false);
ramIndexWriter.setUseCompoundFile(false);

// Here some code for adding documents to RAM index writer
...................................................................................................................
// method where I'm flushing from memory to disk and getting exception
ramIndexWriter.flush();
mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
ramIndexWriter.close();
ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
ramIndexWriter.setUseCompoundFile(false);
System.gc();
{code}

I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.

If you need details on environment(i.e. Java version, settings, hardware details etc.) - please let me know.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-1192) FileNotFound exception on adding a RAMDirectory to an IndexWriter

ASF GitHub Bot (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alex Falca updated LUCENE-1192:
-------------------------------

    Description:
Hi,

I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
{noformat}
java.io.FileNotFoundException: _8.fnm
        at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
        at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
        at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
        at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
        at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
{noformat}

Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
1. First documents are added to and index which use a RAMDirectory as a storage
2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.

Sample code:
{code}
IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
mainIndexWriter.setMergeFactor(10);
mainIndexWriter.setUseCompoundFile(false);
ramIndexWriter.setUseCompoundFile(false);

// Here some code for adding documents to RAM index writer
...................................................................................................................
// method where I'm flushing from memory to disk and getting exception
ramIndexWriter.flush();
mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
ramIndexWriter.close();
ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
ramIndexWriter.setUseCompoundFile(false);
System.gc();
{code}

I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.



  was:
Hi,

I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
{noformat}
java.io.FileNotFoundException: _8.fnm
        at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
        at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
        at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
        at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
        at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
{noformat}

Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
1. First documents are added to and index which use a RAMDirectory as a storage
2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.

Sample code:
{code}
IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
mainIndexWriter.setMergeFactor(10);
mainIndexWriter.setUseCompoundFile(false);
ramIndexWriter.setUseCompoundFile(false);

// Here some code for adding documents to RAM index writer
...................................................................................................................
// method where I'm flushing from memory to disk and getting exception
ramIndexWriter.flush();
mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
ramIndexWriter.close();
ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
ramIndexWriter.setUseCompoundFile(false);
System.gc();
{code}

I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.

If you need details on environment(i.e. Java version, settings, hardware details etc.) - please let me know.

    Environment:
|| Description || Value ||
| java -version | {noformat}java version "1.5.0_14"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)
{noformat} |
| JVM settings | {noformat}-server -verbose:gc -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime
-XX:NewRatio=2 -XX:ParallelGCThreads=4 -Xms6000m -Xmx6000m{noformat} |
| uname -a | {noformat}Linux pdbindexer2 2.6.22.14-72.fc6 #1
SMP Wed Nov 21 14:10:25 EST 2007 x86_64 x86_64 x86_64 GNU/Linux{noformat} |
| CPU | Dual Core AMD Opteron(tm) Processor 280 |
| Total available memory | 8G |

Updated details on environment

> FileNotFound exception on adding a RAMDirectory to an IndexWriter
> -----------------------------------------------------------------
>
>                 Key: LUCENE-1192
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1192
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 2.3.1
>         Environment: || Description || Value ||
> | java -version | {noformat}java version "1.5.0_14"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)
> {noformat} |
> | JVM settings | {noformat}-server -verbose:gc -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime
> -XX:NewRatio=2 -XX:ParallelGCThreads=4 -Xms6000m -Xmx6000m{noformat} |
> | uname -a | {noformat}Linux pdbindexer2 2.6.22.14-72.fc6 #1
> SMP Wed Nov 21 14:10:25 EST 2007 x86_64 x86_64 x86_64 GNU/Linux{noformat} |
> | CPU | Dual Core AMD Opteron(tm) Processor 280 |
> | Total available memory | 8G |
>            Reporter: Alex Falca
>
> Hi,
> I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
> {noformat}
> java.io.FileNotFoundException: _8.fnm
>         at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
>         at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
>         at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
>         at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
>         at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
> {noformat}
> Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
> 1. First documents are added to and index which use a RAMDirectory as a storage
> 2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.
> Sample code:
> {code}
> IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
> IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> mainIndexWriter.setMergeFactor(10);
> mainIndexWriter.setUseCompoundFile(false);
> ramIndexWriter.setUseCompoundFile(false);
> // Here some code for adding documents to RAM index writer
> ...................................................................................................................
> // method where I'm flushing from memory to disk and getting exception
> ramIndexWriter.flush();
> mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
> ramIndexWriter.close();
> ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> ramIndexWriter.setUseCompoundFile(false);
> System.gc();
> {code}
> I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1192) FileNotFound exception on adding a RAMDirectory to an IndexWriter

ASF GitHub Bot (Jira)
In reply to this post by ASF GitHub Bot (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572449#action_12572449 ]

Michael McCandless commented on LUCENE-1192:
--------------------------------------------


This actually isn't quite a valid use of Lucene's APIs, because the
ramIndexWriter is actively changing the files in the RAMDirectory.

This worked in past versions of Lucene because merges were done
synchronously.  Now, with background merges by default, a merge can
complete and commit changes to the RAMDirectory while
addIndexesNoOptimize is running.  The addIndexesNoOptimize method
requires a "static" Directory, where no writer is actively making
changes.

There are a few ways you could get this to work:

  * Sync all concurrent merges before calling addIndexesNoOptimize, by
    calling ((ConcurrentMergeScheduler) ramIndexWriter.getMergeScheduler()).sync()
 
  * Switch back to SerialMergeScheduler

  * Close the ramIndexWriter before passing its directory to
    addIndexesNoOptimize.  In this case you need to separately hold
    the underlying RAMDirectory since you can't call
    IndexWriter.getDirectory after close.



> FileNotFound exception on adding a RAMDirectory to an IndexWriter
> -----------------------------------------------------------------
>
>                 Key: LUCENE-1192
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1192
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 2.3.1
>         Environment: || Description || Value ||
> | java -version | {noformat}java version "1.5.0_14"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)
> {noformat} |
> | JVM settings | {noformat}-server -verbose:gc -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime
> -XX:NewRatio=2 -XX:ParallelGCThreads=4 -Xms6000m -Xmx6000m{noformat} |
> | uname -a | {noformat}Linux pdbindexer2 2.6.22.14-72.fc6 #1
> SMP Wed Nov 21 14:10:25 EST 2007 x86_64 x86_64 x86_64 GNU/Linux{noformat} |
> | CPU | Dual Core AMD Opteron(tm) Processor 280 |
> | Total available memory | 8G |
>            Reporter: Alex Falca
>
> Hi,
> I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
> {noformat}
> java.io.FileNotFoundException: _8.fnm
>         at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
>         at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
>         at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
>         at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
>         at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
> {noformat}
> Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
> 1. First documents are added to and index which use a RAMDirectory as a storage
> 2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.
> Sample code:
> {code}
> IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
> IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> mainIndexWriter.setMergeFactor(10);
> mainIndexWriter.setUseCompoundFile(false);
> ramIndexWriter.setUseCompoundFile(false);
> // Here some code for adding documents to RAM index writer
> ...................................................................................................................
> // method where I'm flushing from memory to disk and getting exception
> ramIndexWriter.flush();
> mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
> ramIndexWriter.close();
> ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> ramIndexWriter.setUseCompoundFile(false);
> System.gc();
> {code}
> I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1192) FileNotFound exception on adding a RAMDirectory to an IndexWriter

ASF GitHub Bot (Jira)
In reply to this post by ASF GitHub Bot (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572457#action_12572457 ]

Alex Falca commented on LUCENE-1192:
------------------------------------

Thanks for the explanations and proposed ways to solve issue:) I'll try one of them today.

> FileNotFound exception on adding a RAMDirectory to an IndexWriter
> -----------------------------------------------------------------
>
>                 Key: LUCENE-1192
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1192
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 2.3.1
>         Environment: || Description || Value ||
> | java -version | {noformat}java version "1.5.0_14"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)
> {noformat} |
> | JVM settings | {noformat}-server -verbose:gc -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime
> -XX:NewRatio=2 -XX:ParallelGCThreads=4 -Xms6000m -Xmx6000m{noformat} |
> | uname -a | {noformat}Linux pdbindexer2 2.6.22.14-72.fc6 #1
> SMP Wed Nov 21 14:10:25 EST 2007 x86_64 x86_64 x86_64 GNU/Linux{noformat} |
> | CPU | Dual Core AMD Opteron(tm) Processor 280 |
> | Total available memory | 8G |
>            Reporter: Alex Falca
>
> Hi,
> I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
> {noformat}
> java.io.FileNotFoundException: _8.fnm
>         at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
>         at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
>         at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
>         at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
>         at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
> {noformat}
> Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
> 1. First documents are added to and index which use a RAMDirectory as a storage
> 2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.
> Sample code:
> {code}
> IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
> IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> mainIndexWriter.setMergeFactor(10);
> mainIndexWriter.setUseCompoundFile(false);
> ramIndexWriter.setUseCompoundFile(false);
> // Here some code for adding documents to RAM index writer
> ...................................................................................................................
> // method where I'm flushing from memory to disk and getting exception
> ramIndexWriter.flush();
> mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
> ramIndexWriter.close();
> ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> ramIndexWriter.setUseCompoundFile(false);
> System.gc();
> {code}
> I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (LUCENE-1192) FileNotFound exception on adding a RAMDirectory to an IndexWriter

ASF GitHub Bot (Jira)
In reply to this post by ASF GitHub Bot (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless resolved LUCENE-1192.
----------------------------------------

    Resolution: Invalid

> FileNotFound exception on adding a RAMDirectory to an IndexWriter
> -----------------------------------------------------------------
>
>                 Key: LUCENE-1192
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1192
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 2.3.1
>         Environment: || Description || Value ||
> | java -version | {noformat}java version "1.5.0_14"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)
> {noformat} |
> | JVM settings | {noformat}-server -verbose:gc -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime
> -XX:NewRatio=2 -XX:ParallelGCThreads=4 -Xms6000m -Xmx6000m{noformat} |
> | uname -a | {noformat}Linux pdbindexer2 2.6.22.14-72.fc6 #1
> SMP Wed Nov 21 14:10:25 EST 2007 x86_64 x86_64 x86_64 GNU/Linux{noformat} |
> | CPU | Dual Core AMD Opteron(tm) Processor 280 |
> | Total available memory | 8G |
>            Reporter: Alex Falca
>
> Hi,
> I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
> {noformat}
> java.io.FileNotFoundException: _8.fnm
>         at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
>         at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
>         at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
>         at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
>         at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
> {noformat}
> Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
> 1. First documents are added to and index which use a RAMDirectory as a storage
> 2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.
> Sample code:
> {code}
> IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
> IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> mainIndexWriter.setMergeFactor(10);
> mainIndexWriter.setUseCompoundFile(false);
> ramIndexWriter.setUseCompoundFile(false);
> // Here some code for adding documents to RAM index writer
> ...................................................................................................................
> // method where I'm flushing from memory to disk and getting exception
> ramIndexWriter.flush();
> mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
> ramIndexWriter.close();
> ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> ramIndexWriter.setUseCompoundFile(false);
> System.gc();
> {code}
> I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1192) FileNotFound exception on adding a RAMDirectory to an IndexWriter

ASF GitHub Bot (Jira)
In reply to this post by ASF GitHub Bot (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572495#action_12572495 ]

Michael McCandless commented on LUCENE-1192:
--------------------------------------------

You're welcome!

> FileNotFound exception on adding a RAMDirectory to an IndexWriter
> -----------------------------------------------------------------
>
>                 Key: LUCENE-1192
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1192
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 2.3.1
>         Environment: || Description || Value ||
> | java -version | {noformat}java version "1.5.0_14"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)
> {noformat} |
> | JVM settings | {noformat}-server -verbose:gc -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime
> -XX:NewRatio=2 -XX:ParallelGCThreads=4 -Xms6000m -Xmx6000m{noformat} |
> | uname -a | {noformat}Linux pdbindexer2 2.6.22.14-72.fc6 #1
> SMP Wed Nov 21 14:10:25 EST 2007 x86_64 x86_64 x86_64 GNU/Linux{noformat} |
> | CPU | Dual Core AMD Opteron(tm) Processor 280 |
> | Total available memory | 8G |
>            Reporter: Alex Falca
>
> Hi,
> I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
> {noformat}
> java.io.FileNotFoundException: _8.fnm
>         at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
>         at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
>         at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
>         at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
>         at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
> {noformat}
> Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
> 1. First documents are added to and index which use a RAMDirectory as a storage
> 2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.
> Sample code:
> {code}
> IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
> IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> mainIndexWriter.setMergeFactor(10);
> mainIndexWriter.setUseCompoundFile(false);
> ramIndexWriter.setUseCompoundFile(false);
> // Here some code for adding documents to RAM index writer
> ...................................................................................................................
> // method where I'm flushing from memory to disk and getting exception
> ramIndexWriter.flush();
> mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
> ramIndexWriter.close();
> ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> ramIndexWriter.setUseCompoundFile(false);
> System.gc();
> {code}
> I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Closed: (LUCENE-1192) FileNotFound exception on adding a RAMDirectory to an IndexWriter

ASF GitHub Bot (Jira)
In reply to this post by ASF GitHub Bot (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alex Falca closed LUCENE-1192.
------------------------------


Tried last option:
{quote}
    *  Close the ramIndexWriter before passing its directory to
      addIndexesNoOptimize. In this case you need to separately hold
      the underlying RAMDirectory since you can't call
      IndexWriter.getDirectory after close.
{quote}

everything works fine :) Got a 2x increase in speed of indexing.
Thank you again!
Closing ticket.

> FileNotFound exception on adding a RAMDirectory to an IndexWriter
> -----------------------------------------------------------------
>
>                 Key: LUCENE-1192
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1192
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 2.3.1
>         Environment: || Description || Value ||
> | java -version | {noformat}java version "1.5.0_14"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)
> {noformat} |
> | JVM settings | {noformat}-server -verbose:gc -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime
> -XX:NewRatio=2 -XX:ParallelGCThreads=4 -Xms6000m -Xmx6000m{noformat} |
> | uname -a | {noformat}Linux pdbindexer2 2.6.22.14-72.fc6 #1
> SMP Wed Nov 21 14:10:25 EST 2007 x86_64 x86_64 x86_64 GNU/Linux{noformat} |
> | CPU | Dual Core AMD Opteron(tm) Processor 280 |
> | Total available memory | 8G |
>            Reporter: Alex Falca
>
> Hi,
> I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
> {noformat}
> java.io.FileNotFoundException: _8.fnm
>         at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
>         at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
>         at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
>         at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
>         at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
> {noformat}
> Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
> 1. First documents are added to and index which use a RAMDirectory as a storage
> 2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.
> Sample code:
> {code}
> IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
> IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> mainIndexWriter.setMergeFactor(10);
> mainIndexWriter.setUseCompoundFile(false);
> ramIndexWriter.setUseCompoundFile(false);
> // Here some code for adding documents to RAM index writer
> ...................................................................................................................
> // method where I'm flushing from memory to disk and getting exception
> ramIndexWriter.flush();
> mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
> ramIndexWriter.close();
> ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> ramIndexWriter.setUseCompoundFile(false);
> System.gc();
> {code}
> I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]