IndexWriter.close() no longer seems to close everything

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

IndexWriter.close() no longer seems to close everything

Daniel Noll-3-2
Hi all.

We updated to Lucene 2.9, and now we find that after closing our text
index, it is not possible to rename the directory in which it resides
(we are actually renaming a directory further up the hierarchy.)

We discovered that the following files were still open by the process:

  _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf

We are calling IndexWriter.close() shortly before attempting to write
to the directory (a few lines of code earlier) so I suspect it could
be related to timing somehow if Lucene is perhaps still doing
something on a background thread at this time (though I was under the
impression that close() waited for merges and so forth to complete
before returning.)

Daniel

--
Daniel Noll                            Forensic and eDiscovery Software
Senior Developer                              The world's most advanced
Nuix                                                email data analysis
http://nuix.com/                                and eDiscovery software

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter.close() no longer seems to close everything

John Wang-9
I am seeing the samething, but only when IndexWriter.getReader is called at
a high rate.

from lsof, I see file handles growing.

-John

On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <[hidden email]> wrote:

> Hi all.
>
> We updated to Lucene 2.9, and now we find that after closing our text
> index, it is not possible to rename the directory in which it resides
> (we are actually renaming a directory further up the hierarchy.)
>
> We discovered that the following files were still open by the process:
>
>  _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>
> We are calling IndexWriter.close() shortly before attempting to write
> to the directory (a few lines of code earlier) so I suspect it could
> be related to timing somehow if Lucene is perhaps still doing
> something on a background thread at this time (though I was under the
> impression that close() waited for merges and so forth to complete
> before returning.)
>
> Daniel
>
> --
> Daniel Noll                            Forensic and eDiscovery Software
> Senior Developer                              The world's most advanced
> Nuix                                                email data analysis
> http://nuix.com/                                and eDiscovery software
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter.close() no longer seems to close everything

Daniel Noll-3-2
On Mon, Nov 9, 2009 at 14:41, John Wang <[hidden email]> wrote:
> I am seeing the samething, but only when IndexWriter.getReader is called at
> a high rate.
>
> from lsof, I see file handles growing.

This hint turned out to help. :-)

Turns out we had an IndexReader hanging around from a previous index
state (before adding docs.)  When the IndexWriter was opened, another
reader was opened, so even though we thought we were closing both, it
turned out there were two readers and one writer, and we were only
closing one of the readers.

Daniel

--
Daniel Noll                            Forensic and eDiscovery Software
Senior Developer                              The world's most advanced
Nuix                                                email data analysis
http://nuix.com/                                and eDiscovery software

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter.close() no longer seems to close everything

Michael McCandless-2
In reply to this post by John Wang-9
Does this look like a real leak John?  You're definitely closing every
reader you get back from getReader?

Mike

On Sun, Nov 8, 2009 at 10:41 PM, John Wang <[hidden email]> wrote:

> I am seeing the samething, but only when IndexWriter.getReader is called at
> a high rate.
>
> from lsof, I see file handles growing.
>
> -John
>
> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <[hidden email]> wrote:
>
>> Hi all.
>>
>> We updated to Lucene 2.9, and now we find that after closing our text
>> index, it is not possible to rename the directory in which it resides
>> (we are actually renaming a directory further up the hierarchy.)
>>
>> We discovered that the following files were still open by the process:
>>
>>  _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>
>> We are calling IndexWriter.close() shortly before attempting to write
>> to the directory (a few lines of code earlier) so I suspect it could
>> be related to timing somehow if Lucene is perhaps still doing
>> something on a background thread at this time (though I was under the
>> impression that close() waited for merges and so forth to complete
>> before returning.)
>>
>> Daniel
>>
>> --
>> Daniel Noll                            Forensic and eDiscovery Software
>> Senior Developer                              The world's most advanced
>> Nuix                                                email data analysis
>> http://nuix.com/                                and eDiscovery software
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter.close() no longer seems to close everything

Albert Juhe
I don't know if it's the same problem but I think it's similar,

My problem is with the Indexsearcher. I've installed a web search engine that uses Lucene, after a search I make a close operation like this:

private IndexSearcher searcher;

NIOFSDirectory directory = new NIOFSDirectory(new File(path));
this.searcher = new IndexSearcher(directory, true);

 public void close() {
        try {        
            System.out.println("Closing: " +this.searcher.getIndexReader().directory().toString() );
            this.searcher.getIndexReader().directory().close();
            this.searcher.getIndexReader().close();
            this.searcher.close();
        } catch (IOException e) {
            System.out.println(" caught a " + e.getClass() + "\n with message: " + e.getMessage());
        }
    }
jboss don't close *.cfs index files, everytime that I make a new search a new file is caught by jboss.
If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command, the only way to free this files and close it is stopping jboss.

The problem is that after search many times,  jboss go down and I have to restart it.

Do you have the same problem with IndexSearch or is only with IndexWriter?

Albert Juhe
Learning Technologies
Universitat Oberta de Catalunya

Michael McCandless-2 wrote
Does this look like a real leak John?  You're definitely closing every
reader you get back from getReader?

Mike

On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang@gmail.com> wrote:
> I am seeing the samething, but only when IndexWriter.getReader is called at
> a high rate.
>
> from lsof, I see file handles growing.
>
> -John
>
> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel@nuix.com> wrote:
>
>> Hi all.
>>
>> We updated to Lucene 2.9, and now we find that after closing our text
>> index, it is not possible to rename the directory in which it resides
>> (we are actually renaming a directory further up the hierarchy.)
>>
>> We discovered that the following files were still open by the process:
>>
>>  _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>
>> We are calling IndexWriter.close() shortly before attempting to write
>> to the directory (a few lines of code earlier) so I suspect it could
>> be related to timing somehow if Lucene is perhaps still doing
>> something on a background thread at this time (though I was under the
>> impression that close() waited for merges and so forth to complete
>> before returning.)
>>
>> Daniel
>>
>> --
>> Daniel Noll                            Forensic and eDiscovery Software
>> Senior Developer                              The world's most advanced
>> Nuix                                                email data analysis
>> http://nuix.com/                                and eDiscovery software
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter.close() no longer seems to close everything

Michael McCandless-2
Do you see your exception handler printing anything out?

You don't need to close the underlying IndexReader, just the
IndexSearcher (which will close the IndexReader, since it was the one
that had opened it).

Mike

On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe <[hidden email]> wrote:

>
> I don't know if it's the same problem but I think it's similar,
>
> My problem is with the Indexsearcher. I've installed a web search engine
> that uses Lucene, after a search I make a close operation like this:
>
> private IndexSearcher searcher;
>
> NIOFSDirectory directory = new NIOFSDirectory(new File(path));
> this.searcher = new IndexSearcher(directory, true);
>
>  public void close() {
>        try {
>            System.out.println("Closing: "
> +this.searcher.getIndexReader().directory().toString() );
>            this.searcher.getIndexReader().directory().close();
>            this.searcher.getIndexReader().close();
>            this.searcher.close();
>        } catch (IOException e) {
>            System.out.println(" caught a " + e.getClass() + "\n with
> message: " + e.getMessage());
>        }
>    }
> jboss don't close *.cfs index files, everytime that I make a new search a
> new file is caught by jboss.
> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command, the
> only way to free this files and close it is stopping jboss.
>
> The problem is that after search many times,  jboss go down and I have to
> restart it.
>
> Do you have the same problem with IndexSearch or is only with IndexWriter?
>
> Albert Juhe
> Learning Technologies
> Universitat Oberta de Catalunya
>
>
> Michael McCandless-2 wrote:
>>
>> Does this look like a real leak John?  You're definitely closing every
>> reader you get back from getReader?
>>
>> Mike
>>
>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <[hidden email]> wrote:
>>> I am seeing the samething, but only when IndexWriter.getReader is called
>>> at
>>> a high rate.
>>>
>>> from lsof, I see file handles growing.
>>>
>>> -John
>>>
>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <[hidden email]> wrote:
>>>
>>>> Hi all.
>>>>
>>>> We updated to Lucene 2.9, and now we find that after closing our text
>>>> index, it is not possible to rename the directory in which it resides
>>>> (we are actually renaming a directory further up the hierarchy.)
>>>>
>>>> We discovered that the following files were still open by the process:
>>>>
>>>>  _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>>>
>>>> We are calling IndexWriter.close() shortly before attempting to write
>>>> to the directory (a few lines of code earlier) so I suspect it could
>>>> be related to timing somehow if Lucene is perhaps still doing
>>>> something on a background thread at this time (though I was under the
>>>> impression that close() waited for merges and so forth to complete
>>>> before returning.)
>>>>
>>>> Daniel
>>>>
>>>> --
>>>> Daniel Noll                            Forensic and eDiscovery Software
>>>> Senior Developer                              The world's most advanced
>>>> Nuix                                                email data analysis
>>>> http://nuix.com/                                and eDiscovery software
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26298910.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter.close() no longer seems to close everything

Albert Juhe
I don't get any exception.

thank you Mike

Michael McCandless-2 wrote
Do you see your exception handler printing anything out?

You don't need to close the underlying IndexReader, just the
IndexSearcher (which will close the IndexReader, since it was the one
that had opened it).

Mike

On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe <albertjuhe@gmail.com> wrote:
>
> I don't know if it's the same problem but I think it's similar,
>
> My problem is with the Indexsearcher. I've installed a web search engine
> that uses Lucene, after a search I make a close operation like this:
>
> private IndexSearcher searcher;
>
> NIOFSDirectory directory = new NIOFSDirectory(new File(path));
> this.searcher = new IndexSearcher(directory, true);
>
>  public void close() {
>        try {
>            System.out.println("Closing: "
> +this.searcher.getIndexReader().directory().toString() );
>            this.searcher.getIndexReader().directory().close();
>            this.searcher.getIndexReader().close();
>            this.searcher.close();
>        } catch (IOException e) {
>            System.out.println(" caught a " + e.getClass() + "\n with
> message: " + e.getMessage());
>        }
>    }
> jboss don't close *.cfs index files, everytime that I make a new search a
> new file is caught by jboss.
> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command, the
> only way to free this files and close it is stopping jboss.
>
> The problem is that after search many times,  jboss go down and I have to
> restart it.
>
> Do you have the same problem with IndexSearch or is only with IndexWriter?
>
> Albert Juhe
> Learning Technologies
> Universitat Oberta de Catalunya
>
>
> Michael McCandless-2 wrote:
>>
>> Does this look like a real leak John?  You're definitely closing every
>> reader you get back from getReader?
>>
>> Mike
>>
>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang@gmail.com> wrote:
>>> I am seeing the samething, but only when IndexWriter.getReader is called
>>> at
>>> a high rate.
>>>
>>> from lsof, I see file handles growing.
>>>
>>> -John
>>>
>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel@nuix.com> wrote:
>>>
>>>> Hi all.
>>>>
>>>> We updated to Lucene 2.9, and now we find that after closing our text
>>>> index, it is not possible to rename the directory in which it resides
>>>> (we are actually renaming a directory further up the hierarchy.)
>>>>
>>>> We discovered that the following files were still open by the process:
>>>>
>>>>  _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>>>
>>>> We are calling IndexWriter.close() shortly before attempting to write
>>>> to the directory (a few lines of code earlier) so I suspect it could
>>>> be related to timing somehow if Lucene is perhaps still doing
>>>> something on a background thread at this time (though I was under the
>>>> impression that close() waited for merges and so forth to complete
>>>> before returning.)
>>>>
>>>> Daniel
>>>>
>>>> --
>>>> Daniel Noll                            Forensic and eDiscovery Software
>>>> Senior Developer                              The world's most advanced
>>>> Nuix                                                email data analysis
>>>> http://nuix.com/                                and eDiscovery software
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26298910.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter.close() no longer seems to close everything

Michael McCandless-2
Can you narrow the leak down to a small self-contained test?

Mike

On Wed, Nov 11, 2009 at 5:54 AM, Albert Juhe <[hidden email]> wrote:

>
> I don't get any exception.
>
> thank you Mike
>
>
> Michael McCandless-2 wrote:
>>
>> Do you see your exception handler printing anything out?
>>
>> You don't need to close the underlying IndexReader, just the
>> IndexSearcher (which will close the IndexReader, since it was the one
>> that had opened it).
>>
>> Mike
>>
>> On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe <[hidden email]> wrote:
>>>
>>> I don't know if it's the same problem but I think it's similar,
>>>
>>> My problem is with the Indexsearcher. I've installed a web search engine
>>> that uses Lucene, after a search I make a close operation like this:
>>>
>>> private IndexSearcher searcher;
>>>
>>> NIOFSDirectory directory = new NIOFSDirectory(new File(path));
>>> this.searcher = new IndexSearcher(directory, true);
>>>
>>>  public void close() {
>>>        try {
>>>            System.out.println("Closing: "
>>> +this.searcher.getIndexReader().directory().toString() );
>>>            this.searcher.getIndexReader().directory().close();
>>>            this.searcher.getIndexReader().close();
>>>            this.searcher.close();
>>>        } catch (IOException e) {
>>>            System.out.println(" caught a " + e.getClass() + "\n with
>>> message: " + e.getMessage());
>>>        }
>>>    }
>>> jboss don't close *.cfs index files, everytime that I make a new search a
>>> new file is caught by jboss.
>>> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command, the
>>> only way to free this files and close it is stopping jboss.
>>>
>>> The problem is that after search many times,  jboss go down and I have to
>>> restart it.
>>>
>>> Do you have the same problem with IndexSearch or is only with
>>> IndexWriter?
>>>
>>> Albert Juhe
>>> Learning Technologies
>>> Universitat Oberta de Catalunya
>>>
>>>
>>> Michael McCandless-2 wrote:
>>>>
>>>> Does this look like a real leak John?  You're definitely closing every
>>>> reader you get back from getReader?
>>>>
>>>> Mike
>>>>
>>>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <[hidden email]> wrote:
>>>>> I am seeing the samething, but only when IndexWriter.getReader is
>>>>> called
>>>>> at
>>>>> a high rate.
>>>>>
>>>>> from lsof, I see file handles growing.
>>>>>
>>>>> -John
>>>>>
>>>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <[hidden email]> wrote:
>>>>>
>>>>>> Hi all.
>>>>>>
>>>>>> We updated to Lucene 2.9, and now we find that after closing our text
>>>>>> index, it is not possible to rename the directory in which it resides
>>>>>> (we are actually renaming a directory further up the hierarchy.)
>>>>>>
>>>>>> We discovered that the following files were still open by the process:
>>>>>>
>>>>>>  _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>>>>>
>>>>>> We are calling IndexWriter.close() shortly before attempting to write
>>>>>> to the directory (a few lines of code earlier) so I suspect it could
>>>>>> be related to timing somehow if Lucene is perhaps still doing
>>>>>> something on a background thread at this time (though I was under the
>>>>>> impression that close() waited for merges and so forth to complete
>>>>>> before returning.)
>>>>>>
>>>>>> Daniel
>>>>>>
>>>>>> --
>>>>>> Daniel Noll                            Forensic and eDiscovery
>>>>>> Software
>>>>>> Senior Developer                              The world's most
>>>>>> advanced
>>>>>> Nuix                                                email data
>>>>>> analysis
>>>>>> http://nuix.com/                                and eDiscovery
>>>>>> software
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26298910.html
>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299405.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter.close() no longer seems to close everything

Albert Juhe
I think that this is the best way to proceed.

thank you Mike


Michael McCandless-2 wrote
Can you narrow the leak down to a small self-contained test?

Mike

On Wed, Nov 11, 2009 at 5:54 AM, Albert Juhe <albertjuhe@gmail.com> wrote:
>
> I don't get any exception.
>
> thank you Mike
>
>
> Michael McCandless-2 wrote:
>>
>> Do you see your exception handler printing anything out?
>>
>> You don't need to close the underlying IndexReader, just the
>> IndexSearcher (which will close the IndexReader, since it was the one
>> that had opened it).
>>
>> Mike
>>
>> On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe <albertjuhe@gmail.com> wrote:
>>>
>>> I don't know if it's the same problem but I think it's similar,
>>>
>>> My problem is with the Indexsearcher. I've installed a web search engine
>>> that uses Lucene, after a search I make a close operation like this:
>>>
>>> private IndexSearcher searcher;
>>>
>>> NIOFSDirectory directory = new NIOFSDirectory(new File(path));
>>> this.searcher = new IndexSearcher(directory, true);
>>>
>>>  public void close() {
>>>        try {
>>>            System.out.println("Closing: "
>>> +this.searcher.getIndexReader().directory().toString() );
>>>            this.searcher.getIndexReader().directory().close();
>>>            this.searcher.getIndexReader().close();
>>>            this.searcher.close();
>>>        } catch (IOException e) {
>>>            System.out.println(" caught a " + e.getClass() + "\n with
>>> message: " + e.getMessage());
>>>        }
>>>    }
>>> jboss don't close *.cfs index files, everytime that I make a new search a
>>> new file is caught by jboss.
>>> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command, the
>>> only way to free this files and close it is stopping jboss.
>>>
>>> The problem is that after search many times,  jboss go down and I have to
>>> restart it.
>>>
>>> Do you have the same problem with IndexSearch or is only with
>>> IndexWriter?
>>>
>>> Albert Juhe
>>> Learning Technologies
>>> Universitat Oberta de Catalunya
>>>
>>>
>>> Michael McCandless-2 wrote:
>>>>
>>>> Does this look like a real leak John?  You're definitely closing every
>>>> reader you get back from getReader?
>>>>
>>>> Mike
>>>>
>>>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang@gmail.com> wrote:
>>>>> I am seeing the samething, but only when IndexWriter.getReader is
>>>>> called
>>>>> at
>>>>> a high rate.
>>>>>
>>>>> from lsof, I see file handles growing.
>>>>>
>>>>> -John
>>>>>
>>>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel@nuix.com> wrote:
>>>>>
>>>>>> Hi all.
>>>>>>
>>>>>> We updated to Lucene 2.9, and now we find that after closing our text
>>>>>> index, it is not possible to rename the directory in which it resides
>>>>>> (we are actually renaming a directory further up the hierarchy.)
>>>>>>
>>>>>> We discovered that the following files were still open by the process:
>>>>>>
>>>>>>  _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>>>>>
>>>>>> We are calling IndexWriter.close() shortly before attempting to write
>>>>>> to the directory (a few lines of code earlier) so I suspect it could
>>>>>> be related to timing somehow if Lucene is perhaps still doing
>>>>>> something on a background thread at this time (though I was under the
>>>>>> impression that close() waited for merges and so forth to complete
>>>>>> before returning.)
>>>>>>
>>>>>> Daniel
>>>>>>
>>>>>> --
>>>>>> Daniel Noll                            Forensic and eDiscovery
>>>>>> Software
>>>>>> Senior Developer                              The world's most
>>>>>> advanced
>>>>>> Nuix                                                email data
>>>>>> analysis
>>>>>> http://nuix.com/                                and eDiscovery
>>>>>> software
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26298910.html
>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299405.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter.close() no longer seems to close everything

John Wang-9
In reply to this post by Michael McCandless-2
Well, I have code in the finally block to call IndexReader.close for every
reader I get from IndexWriter.getReader.

On Mon, Nov 9, 2009 at 2:43 AM, Michael McCandless <
[hidden email]> wrote:

> Does this look like a real leak John?  You're definitely closing every
> reader you get back from getReader?
>
> Mike
>
> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <[hidden email]> wrote:
> > I am seeing the samething, but only when IndexWriter.getReader is called
> at
> > a high rate.
> >
> > from lsof, I see file handles growing.
> >
> > -John
> >
> > On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <[hidden email]> wrote:
> >
> >> Hi all.
> >>
> >> We updated to Lucene 2.9, and now we find that after closing our text
> >> index, it is not possible to rename the directory in which it resides
> >> (we are actually renaming a directory further up the hierarchy.)
> >>
> >> We discovered that the following files were still open by the process:
> >>
> >>  _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
> >>
> >> We are calling IndexWriter.close() shortly before attempting to write
> >> to the directory (a few lines of code earlier) so I suspect it could
> >> be related to timing somehow if Lucene is perhaps still doing
> >> something on a background thread at this time (though I was under the
> >> impression that close() waited for merges and so forth to complete
> >> before returning.)
> >>
> >> Daniel
> >>
> >> --
> >> Daniel Noll                            Forensic and eDiscovery Software
> >> Senior Developer                              The world's most advanced
> >> Nuix                                                email data analysis
> >> http://nuix.com/                                and eDiscovery software
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter.close() no longer seems to close everything

John Wang-9
If you run the zoie test turned to nrt, you can replicate it rather easily:

While the test is running, do lsof on your process, e.g.

lsof -p <proc_id> | <indexdir> | wc

-John

On Thu, Nov 12, 2009 at 8:24 AM, John Wang <[hidden email]> wrote:

> Well, I have code in the finally block to call IndexReader.close for every
> reader I get from IndexWriter.getReader.
>
>
> On Mon, Nov 9, 2009 at 2:43 AM, Michael McCandless <
> [hidden email]> wrote:
>
>> Does this look like a real leak John?  You're definitely closing every
>> reader you get back from getReader?
>>
>> Mike
>>
>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <[hidden email]> wrote:
>> > I am seeing the samething, but only when IndexWriter.getReader is called
>> at
>> > a high rate.
>> >
>> > from lsof, I see file handles growing.
>> >
>> > -John
>> >
>> > On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <[hidden email]> wrote:
>> >
>> >> Hi all.
>> >>
>> >> We updated to Lucene 2.9, and now we find that after closing our text
>> >> index, it is not possible to rename the directory in which it resides
>> >> (we are actually renaming a directory further up the hierarchy.)
>> >>
>> >> We discovered that the following files were still open by the process:
>> >>
>> >>  _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>> >>
>> >> We are calling IndexWriter.close() shortly before attempting to write
>> >> to the directory (a few lines of code earlier) so I suspect it could
>> >> be related to timing somehow if Lucene is perhaps still doing
>> >> something on a background thread at this time (though I was under the
>> >> impression that close() waited for merges and so forth to complete
>> >> before returning.)
>> >>
>> >> Daniel
>> >>
>> >> --
>> >> Daniel Noll                            Forensic and eDiscovery Software
>> >> Senior Developer                              The world's most advanced
>> >> Nuix                                                email data analysis
>> >> http://nuix.com/                                and eDiscovery
>> software
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: [hidden email]
>> >> For additional commands, e-mail: [hidden email]
>> >>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter.close() no longer seems to close everything

Jason Rutherglen
In reply to this post by Albert Juhe
If there's a bug you're seeing, it's helpful to open an issue and post
code reproducing it.

On Wed, Nov 11, 2009 at 3:41 AM, Albert Juhe <[hidden email]> wrote:

>
> I think that this is the best way to proceed.
>
> thank you Mike
>
>
>
> Michael McCandless-2 wrote:
>>
>> Can you narrow the leak down to a small self-contained test?
>>
>> Mike
>>
>> On Wed, Nov 11, 2009 at 5:54 AM, Albert Juhe <[hidden email]> wrote:
>>>
>>> I don't get any exception.
>>>
>>> thank you Mike
>>>
>>>
>>> Michael McCandless-2 wrote:
>>>>
>>>> Do you see your exception handler printing anything out?
>>>>
>>>> You don't need to close the underlying IndexReader, just the
>>>> IndexSearcher (which will close the IndexReader, since it was the one
>>>> that had opened it).
>>>>
>>>> Mike
>>>>
>>>> On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe <[hidden email]>
>>>> wrote:
>>>>>
>>>>> I don't know if it's the same problem but I think it's similar,
>>>>>
>>>>> My problem is with the Indexsearcher. I've installed a web search
>>>>> engine
>>>>> that uses Lucene, after a search I make a close operation like this:
>>>>>
>>>>> private IndexSearcher searcher;
>>>>>
>>>>> NIOFSDirectory directory = new NIOFSDirectory(new File(path));
>>>>> this.searcher = new IndexSearcher(directory, true);
>>>>>
>>>>>  public void close() {
>>>>>        try {
>>>>>            System.out.println("Closing: "
>>>>> +this.searcher.getIndexReader().directory().toString() );
>>>>>            this.searcher.getIndexReader().directory().close();
>>>>>            this.searcher.getIndexReader().close();
>>>>>            this.searcher.close();
>>>>>        } catch (IOException e) {
>>>>>            System.out.println(" caught a " + e.getClass() + "\n with
>>>>> message: " + e.getMessage());
>>>>>        }
>>>>>    }
>>>>> jboss don't close *.cfs index files, everytime that I make a new search
>>>>> a
>>>>> new file is caught by jboss.
>>>>> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command,
>>>>> the
>>>>> only way to free this files and close it is stopping jboss.
>>>>>
>>>>> The problem is that after search many times,  jboss go down and I have
>>>>> to
>>>>> restart it.
>>>>>
>>>>> Do you have the same problem with IndexSearch or is only with
>>>>> IndexWriter?
>>>>>
>>>>> Albert Juhe
>>>>> Learning Technologies
>>>>> Universitat Oberta de Catalunya
>>>>>
>>>>>
>>>>> Michael McCandless-2 wrote:
>>>>>>
>>>>>> Does this look like a real leak John?  You're definitely closing every
>>>>>> reader you get back from getReader?
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <[hidden email]>
>>>>>> wrote:
>>>>>>> I am seeing the samething, but only when IndexWriter.getReader is
>>>>>>> called
>>>>>>> at
>>>>>>> a high rate.
>>>>>>>
>>>>>>> from lsof, I see file handles growing.
>>>>>>>
>>>>>>> -John
>>>>>>>
>>>>>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <[hidden email]> wrote:
>>>>>>>
>>>>>>>> Hi all.
>>>>>>>>
>>>>>>>> We updated to Lucene 2.9, and now we find that after closing our
>>>>>>>> text
>>>>>>>> index, it is not possible to rename the directory in which it
>>>>>>>> resides
>>>>>>>> (we are actually renaming a directory further up the hierarchy.)
>>>>>>>>
>>>>>>>> We discovered that the following files were still open by the
>>>>>>>> process:
>>>>>>>>
>>>>>>>>  _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>>>>>>>
>>>>>>>> We are calling IndexWriter.close() shortly before attempting to
>>>>>>>> write
>>>>>>>> to the directory (a few lines of code earlier) so I suspect it could
>>>>>>>> be related to timing somehow if Lucene is perhaps still doing
>>>>>>>> something on a background thread at this time (though I was under
>>>>>>>> the
>>>>>>>> impression that close() waited for merges and so forth to complete
>>>>>>>> before returning.)
>>>>>>>>
>>>>>>>> Daniel
>>>>>>>>
>>>>>>>> --
>>>>>>>> Daniel Noll                            Forensic and eDiscovery
>>>>>>>> Software
>>>>>>>> Senior Developer                              The world's most
>>>>>>>> advanced
>>>>>>>> Nuix                                                email data
>>>>>>>> analysis
>>>>>>>> http://nuix.com/                                and eDiscovery
>>>>>>>> software
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26298910.html
>>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: [hidden email]
>>>>> For additional commands, e-mail: [hidden email]
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299405.html
>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299941.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter.close() no longer seems to close everything

Albert Juhe
Hi,

About this problem I did a test yesterday, I did a downgrade, I changed versión 2.9.1 to 2.4.1, and the problem has been solved, all the files are closed corretly and JBOSS isn't unstable.

Another problem that we have observed is:

Sometimes, random success, when you try to make a serach the first time don't return result but the second time yes (Lucene 2.9.1. BooleanQuery).
I get this error message:
java.lang.NullPointerException
        at org.apache.jsp.objects.buscar_005fresposta_jsp._jspService(buscar_005fresposta_jsp.java:631)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
        at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
        at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
        at org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
        at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
        at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
        at java.lang.Thread.run(Thread.java:595)

I don't know whats the meanig of it, but buscar_resposta.jsp is the jsp that makes a Search.

With 2.4.1 this problem don't happens.

Jason Rutherglen-2 wrote
If there's a bug you're seeing, it's helpful to open an issue and post
code reproducing it.

On Wed, Nov 11, 2009 at 3:41 AM, Albert Juhe <albertjuhe@gmail.com> wrote:
>
> I think that this is the best way to proceed.
>
> thank you Mike
>
>
>
> Michael McCandless-2 wrote:
>>
>> Can you narrow the leak down to a small self-contained test?
>>
>> Mike
>>
>> On Wed, Nov 11, 2009 at 5:54 AM, Albert Juhe <albertjuhe@gmail.com> wrote:
>>>
>>> I don't get any exception.
>>>
>>> thank you Mike
>>>
>>>
>>> Michael McCandless-2 wrote:
>>>>
>>>> Do you see your exception handler printing anything out?
>>>>
>>>> You don't need to close the underlying IndexReader, just the
>>>> IndexSearcher (which will close the IndexReader, since it was the one
>>>> that had opened it).
>>>>
>>>> Mike
>>>>
>>>> On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe <albertjuhe@gmail.com>
>>>> wrote:
>>>>>
>>>>> I don't know if it's the same problem but I think it's similar,
>>>>>
>>>>> My problem is with the Indexsearcher. I've installed a web search
>>>>> engine
>>>>> that uses Lucene, after a search I make a close operation like this:
>>>>>
>>>>> private IndexSearcher searcher;
>>>>>
>>>>> NIOFSDirectory directory = new NIOFSDirectory(new File(path));
>>>>> this.searcher = new IndexSearcher(directory, true);
>>>>>
>>>>>  public void close() {
>>>>>        try {
>>>>>            System.out.println("Closing: "
>>>>> +this.searcher.getIndexReader().directory().toString() );
>>>>>            this.searcher.getIndexReader().directory().close();
>>>>>            this.searcher.getIndexReader().close();
>>>>>            this.searcher.close();
>>>>>        } catch (IOException e) {
>>>>>            System.out.println(" caught a " + e.getClass() + "\n with
>>>>> message: " + e.getMessage());
>>>>>        }
>>>>>    }
>>>>> jboss don't close *.cfs index files, everytime that I make a new search
>>>>> a
>>>>> new file is caught by jboss.
>>>>> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command,
>>>>> the
>>>>> only way to free this files and close it is stopping jboss.
>>>>>
>>>>> The problem is that after search many times,  jboss go down and I have
>>>>> to
>>>>> restart it.
>>>>>
>>>>> Do you have the same problem with IndexSearch or is only with
>>>>> IndexWriter?
>>>>>
>>>>> Albert Juhe
>>>>> Learning Technologies
>>>>> Universitat Oberta de Catalunya
>>>>>
>>>>>
>>>>> Michael McCandless-2 wrote:
>>>>>>
>>>>>> Does this look like a real leak John?  You're definitely closing every
>>>>>> reader you get back from getReader?
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang@gmail.com>
>>>>>> wrote:
>>>>>>> I am seeing the samething, but only when IndexWriter.getReader is
>>>>>>> called
>>>>>>> at
>>>>>>> a high rate.
>>>>>>>
>>>>>>> from lsof, I see file handles growing.
>>>>>>>
>>>>>>> -John
>>>>>>>
>>>>>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel@nuix.com> wrote:
>>>>>>>
>>>>>>>> Hi all.
>>>>>>>>
>>>>>>>> We updated to Lucene 2.9, and now we find that after closing our
>>>>>>>> text
>>>>>>>> index, it is not possible to rename the directory in which it
>>>>>>>> resides
>>>>>>>> (we are actually renaming a directory further up the hierarchy.)
>>>>>>>>
>>>>>>>> We discovered that the following files were still open by the
>>>>>>>> process:
>>>>>>>>
>>>>>>>>  _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>>>>>>>
>>>>>>>> We are calling IndexWriter.close() shortly before attempting to
>>>>>>>> write
>>>>>>>> to the directory (a few lines of code earlier) so I suspect it could
>>>>>>>> be related to timing somehow if Lucene is perhaps still doing
>>>>>>>> something on a background thread at this time (though I was under
>>>>>>>> the
>>>>>>>> impression that close() waited for merges and so forth to complete
>>>>>>>> before returning.)
>>>>>>>>
>>>>>>>> Daniel
>>>>>>>>
>>>>>>>> --
>>>>>>>> Daniel Noll                            Forensic and eDiscovery
>>>>>>>> Software
>>>>>>>> Senior Developer                              The world's most
>>>>>>>> advanced
>>>>>>>> Nuix                                                email data
>>>>>>>> analysis
>>>>>>>> http://nuix.com/                                and eDiscovery
>>>>>>>> software
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26298910.html
>>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299405.html
>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299941.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter.close() no longer seems to close everything

Michael McCandless-2
Any luck narrowing this to a standalone test case that shows the problem?

That new exception appears to be inside the Java code created by the
app server compiling your JSP -- it's not very helpful since it
doesn't "enter" Lucene.  Can you try to narrow this to a standalone
test case, too?

Thanks.

Mike

On Fri, Nov 13, 2009 at 5:12 AM, Albert Juhe <[hidden email]> wrote:

>
> Hi,
>
> About this problem I did a test yesterday, I did a downgrade, I changed
> versión 2.9.1 to 2.4.1, and the problem has been solved, all the files are
> closed corretly and JBOSS isn't unstable.
>
> Another problem that we have observed is:
>
> Sometimes, random success, when you try to make a serach the first time
> don't return result but the second time yes (Lucene 2.9.1. BooleanQuery).
> I get this error message:
> java.lang.NullPointerException
>        at
> org.apache.jsp.objects.buscar_005fresposta_jsp._jspService(buscar_005fresposta_jsp.java:631)
>        at
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
>        at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334)
>        at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
>        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
>        at
> org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
>        at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
>        at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
>        at
> org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
>        at
> org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
>        at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
>        at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
>        at
> org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
>        at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
>        at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
>        at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
>        at
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
>        at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
>        at
> org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
>        at java.lang.Thread.run(Thread.java:595)
>
> I don't know whats the meanig of it, but buscar_resposta.jsp is the jsp that
> makes a Search.
>
> With 2.4.1 this problem don't happens.
>
>
> Jason Rutherglen-2 wrote:
>>
>> If there's a bug you're seeing, it's helpful to open an issue and post
>> code reproducing it.
>>
>> On Wed, Nov 11, 2009 at 3:41 AM, Albert Juhe <[hidden email]> wrote:
>>>
>>> I think that this is the best way to proceed.
>>>
>>> thank you Mike
>>>
>>>
>>>
>>> Michael McCandless-2 wrote:
>>>>
>>>> Can you narrow the leak down to a small self-contained test?
>>>>
>>>> Mike
>>>>
>>>> On Wed, Nov 11, 2009 at 5:54 AM, Albert Juhe <[hidden email]>
>>>> wrote:
>>>>>
>>>>> I don't get any exception.
>>>>>
>>>>> thank you Mike
>>>>>
>>>>>
>>>>> Michael McCandless-2 wrote:
>>>>>>
>>>>>> Do you see your exception handler printing anything out?
>>>>>>
>>>>>> You don't need to close the underlying IndexReader, just the
>>>>>> IndexSearcher (which will close the IndexReader, since it was the one
>>>>>> that had opened it).
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe <[hidden email]>
>>>>>> wrote:
>>>>>>>
>>>>>>> I don't know if it's the same problem but I think it's similar,
>>>>>>>
>>>>>>> My problem is with the Indexsearcher. I've installed a web search
>>>>>>> engine
>>>>>>> that uses Lucene, after a search I make a close operation like this:
>>>>>>>
>>>>>>> private IndexSearcher searcher;
>>>>>>>
>>>>>>> NIOFSDirectory directory = new NIOFSDirectory(new File(path));
>>>>>>> this.searcher = new IndexSearcher(directory, true);
>>>>>>>
>>>>>>>  public void close() {
>>>>>>>        try {
>>>>>>>            System.out.println("Closing: "
>>>>>>> +this.searcher.getIndexReader().directory().toString() );
>>>>>>>            this.searcher.getIndexReader().directory().close();
>>>>>>>            this.searcher.getIndexReader().close();
>>>>>>>            this.searcher.close();
>>>>>>>        } catch (IOException e) {
>>>>>>>            System.out.println(" caught a " + e.getClass() + "\n with
>>>>>>> message: " + e.getMessage());
>>>>>>>        }
>>>>>>>    }
>>>>>>> jboss don't close *.cfs index files, everytime that I make a new
>>>>>>> search
>>>>>>> a
>>>>>>> new file is caught by jboss.
>>>>>>> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command,
>>>>>>> the
>>>>>>> only way to free this files and close it is stopping jboss.
>>>>>>>
>>>>>>> The problem is that after search many times,  jboss go down and I
>>>>>>> have
>>>>>>> to
>>>>>>> restart it.
>>>>>>>
>>>>>>> Do you have the same problem with IndexSearch or is only with
>>>>>>> IndexWriter?
>>>>>>>
>>>>>>> Albert Juhe
>>>>>>> Learning Technologies
>>>>>>> Universitat Oberta de Catalunya
>>>>>>>
>>>>>>>
>>>>>>> Michael McCandless-2 wrote:
>>>>>>>>
>>>>>>>> Does this look like a real leak John?  You're definitely closing
>>>>>>>> every
>>>>>>>> reader you get back from getReader?
>>>>>>>>
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <[hidden email]>
>>>>>>>> wrote:
>>>>>>>>> I am seeing the samething, but only when IndexWriter.getReader is
>>>>>>>>> called
>>>>>>>>> at
>>>>>>>>> a high rate.
>>>>>>>>>
>>>>>>>>> from lsof, I see file handles growing.
>>>>>>>>>
>>>>>>>>> -John
>>>>>>>>>
>>>>>>>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <[hidden email]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all.
>>>>>>>>>>
>>>>>>>>>> We updated to Lucene 2.9, and now we find that after closing our
>>>>>>>>>> text
>>>>>>>>>> index, it is not possible to rename the directory in which it
>>>>>>>>>> resides
>>>>>>>>>> (we are actually renaming a directory further up the hierarchy.)
>>>>>>>>>>
>>>>>>>>>> We discovered that the following files were still open by the
>>>>>>>>>> process:
>>>>>>>>>>
>>>>>>>>>>  _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>>>>>>>>>
>>>>>>>>>> We are calling IndexWriter.close() shortly before attempting to
>>>>>>>>>> write
>>>>>>>>>> to the directory (a few lines of code earlier) so I suspect it
>>>>>>>>>> could
>>>>>>>>>> be related to timing somehow if Lucene is perhaps still doing
>>>>>>>>>> something on a background thread at this time (though I was under
>>>>>>>>>> the
>>>>>>>>>> impression that close() waited for merges and so forth to complete
>>>>>>>>>> before returning.)
>>>>>>>>>>
>>>>>>>>>> Daniel
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Daniel Noll                            Forensic and eDiscovery
>>>>>>>>>> Software
>>>>>>>>>> Senior Developer                              The world's most
>>>>>>>>>> advanced
>>>>>>>>>> Nuix                                                email data
>>>>>>>>>> analysis
>>>>>>>>>> http://nuix.com/                                and eDiscovery
>>>>>>>>>> software
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26298910.html
>>>>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299405.html
>>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: [hidden email]
>>>>> For additional commands, e-mail: [hidden email]
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299941.html
>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26334253.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter.close() no longer seems to close everything

Michael McCandless-2
In reply to this post by John Wang-9
Alas, I can't repro this problem ("leaking file descriptors with NRT"), either.

I've got a decent stress test setup -- start with a 5M Wikipedia
index, update (delete & add) @ 1000 docs/sec (using 2 threads), reopen
10X per second, searching at redline (using 9 threads), and the open
file descriptors look contained.

Are you using compound format or not?  Do you hit any odd exceptions
in your test?  Any other differences in how Zoie uses Lucene NRT that
you can think of...?

Mike

On Thu, Nov 12, 2009 at 11:24 AM, John Wang <[hidden email]> wrote:

> Well, I have code in the finally block to call IndexReader.close for every
> reader I get from IndexWriter.getReader.
>
> On Mon, Nov 9, 2009 at 2:43 AM, Michael McCandless <
> [hidden email]> wrote:
>
>> Does this look like a real leak John?  You're definitely closing every
>> reader you get back from getReader?
>>
>> Mike
>>
>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <[hidden email]> wrote:
>> > I am seeing the samething, but only when IndexWriter.getReader is called
>> at
>> > a high rate.
>> >
>> > from lsof, I see file handles growing.
>> >
>> > -John
>> >
>> > On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <[hidden email]> wrote:
>> >
>> >> Hi all.
>> >>
>> >> We updated to Lucene 2.9, and now we find that after closing our text
>> >> index, it is not possible to rename the directory in which it resides
>> >> (we are actually renaming a directory further up the hierarchy.)
>> >>
>> >> We discovered that the following files were still open by the process:
>> >>
>> >>  _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>> >>
>> >> We are calling IndexWriter.close() shortly before attempting to write
>> >> to the directory (a few lines of code earlier) so I suspect it could
>> >> be related to timing somehow if Lucene is perhaps still doing
>> >> something on a background thread at this time (though I was under the
>> >> impression that close() waited for merges and so forth to complete
>> >> before returning.)
>> >>
>> >> Daniel
>> >>
>> >> --
>> >> Daniel Noll                            Forensic and eDiscovery Software
>> >> Senior Developer                              The world's most advanced
>> >> Nuix                                                email data analysis
>> >> http://nuix.com/                                and eDiscovery software
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: [hidden email]
>> >> For additional commands, e-mail: [hidden email]
>> >>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter.close() no longer seems to close everything

Albert Juhe
In reply to this post by Michael McCandless-2
I don't know how, because the problem is with JBOSS in a productions environment, in a localhost this don't happens.

The JBOSS server is in a production environment, contains a lot of projects, I don't know if Lucene enters in fight with othe libraries.

I don't have control of this computer, I send the war to the production departament and they install it, I will try to recover more logs files.

Other part of log file:
2009-11-11 16:50:28,567 INFO  [STDOUT]  ************ START SEARCHING BIBLIOGRAPHY ***************
2009-11-11 16:50:28,577 INFO  [STDOUT] Query MultiFieldQueryParser:tecnologia 1
2009-11-11 16:50:28,577 INFO  [STDOUT] Query MultiFieldQueryParser:tecnologia 1
2009-11-11 16:50:28,577 INFO  [STDOUT] Query MultiFieldQueryParser:tecnologia 1
2009-11-11 16:50:28,731 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/webapps/uocle].[jsp]] Servlet.service() for servlet jsp threw exception

The code that generates the error is:

The query is: MultiFieldQueryParser:tecnologia

BooleanQuery bqCodi = new BooleanQuery();

MultiFieldQueryParser mFilefQueryParser = new MultiFieldQueryParser(this.fields,this.analyzer);            
mFilefQueryParser.setDefaultOperator(QueryParser.AND_OPERATOR);
queryM  = mFilefQueryParser.parse(this.query_text);
bqCodi.add(queryM, BooleanClause.Occur.SHOULD);

DuplicateFilter df = new DuplicateFilter("title");
df.setKeepMode(DuplicateFilter.KM_USE_FIRST_OCCURRENCE);
this.topDocs = getSearcher().search(bqCodi, df, MAX_TERMES_BIBLIOGRAFIA, Sort.RELEVANCE);

I don't know if this information can be usefull.

When I changed the Lucene library to 2.4.1 with the same code, only I have changed how I open Indexreader, in 2.9.1 I opened in readonly=true, the applications runs perfect.

Michael McCandless-2 wrote
Any luck narrowing this to a standalone test case that shows the problem?

That new exception appears to be inside the Java code created by the
app server compiling your JSP -- it's not very helpful since it
doesn't "enter" Lucene.  Can you try to narrow this to a standalone
test case, too?

Thanks.

Mike

On Fri, Nov 13, 2009 at 5:12 AM, Albert Juhe <albertjuhe@gmail.com> wrote:
>
> Hi,
>
> About this problem I did a test yesterday, I did a downgrade, I changed
> versión 2.9.1 to 2.4.1, and the problem has been solved, all the files are
> closed corretly and JBOSS isn't unstable.
>
> Another problem that we have observed is:
>
> Sometimes, random success, when you try to make a serach the first time
> don't return result but the second time yes (Lucene 2.9.1. BooleanQuery).
> I get this error message:
> java.lang.NullPointerException
>        at
> org.apache.jsp.objects.buscar_005fresposta_jsp._jspService(buscar_005fresposta_jsp.java:631)
>        at
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
>        at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334)
>        at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
>        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
>        at
> org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
>        at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
>        at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
>        at
> org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
>        at
> org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
>        at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
>        at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
>        at
> org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
>        at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
>        at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
>        at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
>        at
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
>        at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
>        at
> org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
>        at java.lang.Thread.run(Thread.java:595)
>
> I don't know whats the meanig of it, but buscar_resposta.jsp is the jsp that
> makes a Search.
>
> With 2.4.1 this problem don't happens.
>
>
> Jason Rutherglen-2 wrote:
>>
>> If there's a bug you're seeing, it's helpful to open an issue and post
>> code reproducing it.
>>
>> On Wed, Nov 11, 2009 at 3:41 AM, Albert Juhe <albertjuhe@gmail.com> wrote:
>>>
>>> I think that this is the best way to proceed.
>>>
>>> thank you Mike
>>>
>>>
>>>
>>> Michael McCandless-2 wrote:
>>>>
>>>> Can you narrow the leak down to a small self-contained test?
>>>>
>>>> Mike
>>>>
>>>> On Wed, Nov 11, 2009 at 5:54 AM, Albert Juhe <albertjuhe@gmail.com>
>>>> wrote:
>>>>>
>>>>> I don't get any exception.
>>>>>
>>>>> thank you Mike
>>>>>
>>>>>
>>>>> Michael McCandless-2 wrote:
>>>>>>
>>>>>> Do you see your exception handler printing anything out?
>>>>>>
>>>>>> You don't need to close the underlying IndexReader, just the
>>>>>> IndexSearcher (which will close the IndexReader, since it was the one
>>>>>> that had opened it).
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe <albertjuhe@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> I don't know if it's the same problem but I think it's similar,
>>>>>>>
>>>>>>> My problem is with the Indexsearcher. I've installed a web search
>>>>>>> engine
>>>>>>> that uses Lucene, after a search I make a close operation like this:
>>>>>>>
>>>>>>> private IndexSearcher searcher;
>>>>>>>
>>>>>>> NIOFSDirectory directory = new NIOFSDirectory(new File(path));
>>>>>>> this.searcher = new IndexSearcher(directory, true);
>>>>>>>
>>>>>>>  public void close() {
>>>>>>>        try {
>>>>>>>            System.out.println("Closing: "
>>>>>>> +this.searcher.getIndexReader().directory().toString() );
>>>>>>>            this.searcher.getIndexReader().directory().close();
>>>>>>>            this.searcher.getIndexReader().close();
>>>>>>>            this.searcher.close();
>>>>>>>        } catch (IOException e) {
>>>>>>>            System.out.println(" caught a " + e.getClass() + "\n with
>>>>>>> message: " + e.getMessage());
>>>>>>>        }
>>>>>>>    }
>>>>>>> jboss don't close *.cfs index files, everytime that I make a new
>>>>>>> search
>>>>>>> a
>>>>>>> new file is caught by jboss.
>>>>>>> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command,
>>>>>>> the
>>>>>>> only way to free this files and close it is stopping jboss.
>>>>>>>
>>>>>>> The problem is that after search many times,  jboss go down and I
>>>>>>> have
>>>>>>> to
>>>>>>> restart it.
>>>>>>>
>>>>>>> Do you have the same problem with IndexSearch or is only with
>>>>>>> IndexWriter?
>>>>>>>
>>>>>>> Albert Juhe
>>>>>>> Learning Technologies
>>>>>>> Universitat Oberta de Catalunya
>>>>>>>
>>>>>>>
>>>>>>> Michael McCandless-2 wrote:
>>>>>>>>
>>>>>>>> Does this look like a real leak John?  You're definitely closing
>>>>>>>> every
>>>>>>>> reader you get back from getReader?
>>>>>>>>
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang@gmail.com>
>>>>>>>> wrote:
>>>>>>>>> I am seeing the samething, but only when IndexWriter.getReader is
>>>>>>>>> called
>>>>>>>>> at
>>>>>>>>> a high rate.
>>>>>>>>>
>>>>>>>>> from lsof, I see file handles growing.
>>>>>>>>>
>>>>>>>>> -John
>>>>>>>>>
>>>>>>>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel@nuix.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all.
>>>>>>>>>>
>>>>>>>>>> We updated to Lucene 2.9, and now we find that after closing our
>>>>>>>>>> text
>>>>>>>>>> index, it is not possible to rename the directory in which it
>>>>>>>>>> resides
>>>>>>>>>> (we are actually renaming a directory further up the hierarchy.)
>>>>>>>>>>
>>>>>>>>>> We discovered that the following files were still open by the
>>>>>>>>>> process:
>>>>>>>>>>
>>>>>>>>>>  _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>>>>>>>>>
>>>>>>>>>> We are calling IndexWriter.close() shortly before attempting to
>>>>>>>>>> write
>>>>>>>>>> to the directory (a few lines of code earlier) so I suspect it
>>>>>>>>>> could
>>>>>>>>>> be related to timing somehow if Lucene is perhaps still doing
>>>>>>>>>> something on a background thread at this time (though I was under
>>>>>>>>>> the
>>>>>>>>>> impression that close() waited for merges and so forth to complete
>>>>>>>>>> before returning.)
>>>>>>>>>>
>>>>>>>>>> Daniel
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Daniel Noll                            Forensic and eDiscovery
>>>>>>>>>> Software
>>>>>>>>>> Senior Developer                              The world's most
>>>>>>>>>> advanced
>>>>>>>>>> Nuix                                                email data
>>>>>>>>>> analysis
>>>>>>>>>> http://nuix.com/                                and eDiscovery
>>>>>>>>>> software
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26298910.html
>>>>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299405.html
>>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299941.html
>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26334253.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org