[jira] Created: (LUCENE-481) IndexReader.getCurrentVersion() and isCurrent should use commit lock.

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (LUCENE-481) IndexReader.getCurrentVersion() and isCurrent should use commit lock.

JIRA jira@apache.org
IndexReader.getCurrentVersion() and isCurrent should use commit lock.
---------------------------------------------------------------------

         Key: LUCENE-481
         URL: http://issues.apache.org/jira/browse/LUCENE-481
     Project: Lucene - Java
        Type: Bug
  Components: Index  
    Versions: CVS Nightly - Specify date in submission    
 Environment: Windows platform, all Lucene versions
    Reporter: Luc Vanlerberghe


There is a race condition if one machine is checking the current version of an index while another wants to update the segments file in IndexWriter.close().

java.io.IOException: Cannot delete segments
        at org.apache.lucene.store.FSDirectory.renameFile(FSDirectory.java:213)
        at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:90)
        at org.apache.lucene.index.IndexWriter$3.doBody(IndexWriter.java:503)
        at org.apache.lucene.store.Lock$With.run(Lock.java:109)
        at org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:501)
        at org.apache.lucene.index.IndexWriter.flushRamSegments(IndexWriter.java:440)
        at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:242)

On the windows platform reading the contents of a file disallows deleting the file.

I use Lucene to maintain an index of +-700.000 documents, one server adds documents, while other servers handle the searches.
The search servers poll the index version regularly to check if they have to reopen their IndexSearcher.
Once in a while (about once every two days on average), IndexWriter.close() fails because it cannot delete the previous segments file, even though it hold the commit lock.
The reason is probably that search servers are reading the segments file to check the version without using the commit lock.


--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-481) IndexReader.getCurrentVersion() and isCurrent should use commit lock.

JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/LUCENE-481?page=all ]

Luc Vanlerberghe updated LUCENE-481:
------------------------------------

    Attachment: LUCENE-481.patch

This should fix the problem.

I didn't use the Lock.With helper class to avoid having to wrap the long or boolean result in a Long or Boolean first.
The try ... finally block should be at least as clear to read as the Lock.With construct in this case...

The problem is not limited to Windows Platforms:
The search servers also threw "java.io.FileNotFoundException: ...\segments (The system cannot find the file specified)" once every few days....

This is because the new segments data is first written to the file "segments.new" which is then renamed to "segments".
The javadoc for Directory says renameTo() should be atomic, but the current implementation in FSDirectory isn't.
If the searcher happens to try to read the segments file in the (admittedly very small) time between the "nu.delete()" and the "old.renameTo(nu)" it gets the FileNotFoundException

> IndexReader.getCurrentVersion() and isCurrent should use commit lock.
> ---------------------------------------------------------------------
>
>          Key: LUCENE-481
>          URL: http://issues.apache.org/jira/browse/LUCENE-481
>      Project: Lucene - Java
>         Type: Bug
>   Components: Index
>     Versions: CVS Nightly - Specify date in submission
>  Environment: Windows platform, all Lucene versions
>     Reporter: Luc Vanlerberghe
>  Attachments: LUCENE-481.patch
>
> There is a race condition if one machine is checking the current version of an index while another wants to update the segments file in IndexWriter.close().
> java.io.IOException: Cannot delete segments
> at org.apache.lucene.store.FSDirectory.renameFile(FSDirectory.java:213)
> at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:90)
> at org.apache.lucene.index.IndexWriter$3.doBody(IndexWriter.java:503)
> at org.apache.lucene.store.Lock$With.run(Lock.java:109)
> at org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:501)
> at org.apache.lucene.index.IndexWriter.flushRamSegments(IndexWriter.java:440)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:242)
> On the windows platform reading the contents of a file disallows deleting the file.
> I use Lucene to maintain an index of +-700.000 documents, one server adds documents, while other servers handle the searches.
> The search servers poll the index version regularly to check if they have to reopen their IndexSearcher.
> Once in a while (about once every two days on average), IndexWriter.close() fails because it cannot delete the previous segments file, even though it hold the commit lock.
> The reason is probably that search servers are reading the segments file to check the version without using the commit lock.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (LUCENE-481) IndexReader.getCurrentVersion() and isCurrent should use commit lock.

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/LUCENE-481?page=all ]
     
Yonik Seeley resolved LUCENE-481:
---------------------------------

    Fix Version: 1.9
     Resolution: Fixed

Thanks Luc, i've commited this!

> IndexReader.getCurrentVersion() and isCurrent should use commit lock.
> ---------------------------------------------------------------------
>
>          Key: LUCENE-481
>          URL: http://issues.apache.org/jira/browse/LUCENE-481
>      Project: Lucene - Java
>         Type: Bug
>   Components: Index
>     Versions: CVS Nightly - Specify date in submission
>  Environment: Windows platform, all Lucene versions
>     Reporter: Luc Vanlerberghe
>      Fix For: 1.9
>  Attachments: LUCENE-481.patch
>
> There is a race condition if one machine is checking the current version of an index while another wants to update the segments file in IndexWriter.close().
> java.io.IOException: Cannot delete segments
> at org.apache.lucene.store.FSDirectory.renameFile(FSDirectory.java:213)
> at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:90)
> at org.apache.lucene.index.IndexWriter$3.doBody(IndexWriter.java:503)
> at org.apache.lucene.store.Lock$With.run(Lock.java:109)
> at org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:501)
> at org.apache.lucene.index.IndexWriter.flushRamSegments(IndexWriter.java:440)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:242)
> On the windows platform reading the contents of a file disallows deleting the file.
> I use Lucene to maintain an index of +-700.000 documents, one server adds documents, while other servers handle the searches.
> The search servers poll the index version regularly to check if they have to reopen their IndexSearcher.
> Once in a while (about once every two days on average), IndexWriter.close() fails because it cannot delete the previous segments file, even though it hold the commit lock.
> The reason is probably that search servers are reading the segments file to check the version without using the commit lock.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]