10Gb of .nfsXXX files about a week old in NFS based index directory

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

10Gb of .nfsXXX files about a week old in NFS based index directory

David Loeng
Hi,

We have a customer using lucene on an NFS directory, which contains  
~10Gb of .nfsXXXX files. These files are the means by which NFS  
implements delete-on-close semantics (that is, if the index writer  
commits a delete of a file that is still held open by an index  
reader, the file is renamed to .nfsXXXXX).

The version of Lucene in use is 2.2.

Our IndexSearcher is refreshed almost every minute (new IndexSearcher
(Directory directory)) depending on whether content has been added/
updated/deleted.

What must be done to guarantee that lucene does not hold onto files  
that have been deleted?

Can a dereferenced IndexSearcher (closeReader=true) which has not  
been close()-ed cause lucene to be holding onto file handles of  
deleted index files?

Cheers,
Dave









---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: 10Gb of .nfsXXX files about a week old in NFS based index directory

Michael McCandless-2

David Loeng wrote:

> Hi,
>
> We have a customer using lucene on an NFS directory, which contains  
> ~10Gb of .nfsXXXX files. These files are the means by which NFS  
> implements delete-on-close semantics (that is, if the index writer  
> commits a delete of a file that is still held open by an index  
> reader, the file is renamed to .nfsXXXXX).

As long as the held-open file as well as the delete take place from  
the same machine, then those .nfsXXXX files will be created.

> The version of Lucene in use is 2.2.
>
> Our IndexSearcher is refreshed almost every minute (new  
> IndexSearcher(Directory directory)) depending on whether content has  
> been added/updated/deleted.
>
> What must be done to guarantee that lucene does not hold onto files  
> that have been deleted?
>
> Can a dereferenced IndexSearcher (closeReader=true) which has not  
> been close()-ed cause lucene to be holding onto file handles of  
> deleted index files?

You really should call close() on any IndexSearcher you created.

Lucene's FSDirectory.FSIndexInput.Descriptor class does have a  
finalizer that calls close, however, it's not a good idea to rely on  
that.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]