Lucene Memory Leak

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Lucene Memory Leak

Andy33
I have a memory leak in my lucene search code. I am able to run a few queries fine, but I eventually run out of memory. Please note that I do close and set to null the ivIndexSearcher object elsewhere. Here is the code I am using...


private synchronized Hits doQuery(String field, String queryStr, Sort sortOrder, String indexDirectory) throws Exception
    {
        Directory directory = null;
       
        try
        {
            Analyzer analyzer = new StandardAnalyzer();
           
            directory = FSDirectory.getDirectory(indexDirectory);
           
            //search the index
            ivIndexSearcher = new IndexSearcher(directory);
               
            QueryParser parser = new QueryParser(field, analyzer);
            Query query = parser.parse(queryStr);
            Hits results = ivIndexSearcher.search(query, sortOrder);
           
            return results;
        }
        finally
        {  
            if(null != directory)
            {
                directory.close();
            }
            directory = null;
        }
    }
Reply | Threaded
Open this post in threaded view
|

Re: Lucene Memory Leak

Andy33
Reply | Threaded
Open this post in threaded view
|

Re: Lucene Memory Leak

Mark Miller-3
In reply to this post by Andy33
You should really close the IndexSearcher rather than the directory.

Andy33 wrote:

> I have a memory leak in my lucene search code. I am able to run a few queries
> fine, but I eventually run out of memory. Please note that I do close and
> set to null the ivIndexSearcher object elsewhere. Here is the code I am
> using...
>
>
> private synchronized Hits doQuery(String field, String queryStr, Sort
> sortOrder, String indexDirectory) throws Exception
>     {
> Directory directory = null;
>
> try
> {
>    Analyzer analyzer = new StandardAnalyzer();
>    
>    directory = FSDirectory.getDirectory(indexDirectory);
>    
>    //search the index
>    ivIndexSearcher = new IndexSearcher(directory);
>      
>    QueryParser parser = new QueryParser(field, analyzer);
>    Query query = parser.parse(queryStr);
>    Hits results = ivIndexSearcher.search(query, sortOrder);
>    
>    return results;
> }
> finally
> {  
>    if(null != directory)
>    {
> directory.close();
>    }
>    directory = null;
> }
>     }
>  


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene Memory Leak

Andy33
As stated in my original message, I am closing the IndexSearcher elsewhere. I don't close it in the method I copied because otherwise I lose access to the Hits that come back.


<quote author="markrmiller">
You should really close the IndexSearcher rather than the directory.

Andy33 wrote:
> I have a memory leak in my lucene search code. I am able to run a few queries
> fine, but I eventually run out of memory. Please note that I do close and
> set to null the ivIndexSearcher object elsewhere. Here is the code I am
> using...
Reply | Threaded
Open this post in threaded view
|

Re: Lucene Memory Leak

Grant Ingersoll-2
In reply to this post by Andy33
Closing the directory seems a bit strange, why are you doing that  
(other than it is a public method), especially since you say you are  
keeping the IndexSearcher around?  Also, you probably shouldn't open a  
new searcher every time.  Are your queries on different directories  
every time?

What testing have you done to date that lead you to believe this  
section of code is the cause of the problem?

-Grant

On Sep 2, 2008, at 3:53 PM, Andy33 wrote:

>
> I have a memory leak in my lucene search code. I am able to run a  
> few queries
> fine, but I eventually run out of memory. Please note that I do  
> close and
> set to null the ivIndexSearcher object elsewhere. Here is the code I  
> am
> using...
>
>
> private synchronized Hits doQuery(String field, String queryStr, Sort
> sortOrder, String indexDirectory) throws Exception
>    {
> Directory directory = null;
>
> try
> {
>    Analyzer analyzer = new StandardAnalyzer();
>
>    directory = FSDirectory.getDirectory(indexDirectory);
>
>    //search the index
>    ivIndexSearcher = new IndexSearcher(directory);
>
>    QueryParser parser = new QueryParser(field, analyzer);
>    Query query = parser.parse(queryStr);
>    Hits results = ivIndexSearcher.search(query, sortOrder);
>
>    return results;
> }
> finally
> {
>    if(null != directory)
>    {
> directory.close();
>    }
>    directory = null;
> }
>    }
> --
> View this message in context: http://www.nabble.com/Lucene-Memory-Leak-tp19276999p19276999.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

--------------------------
Grant Ingersoll
http://www.lucidimagination.com

Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ








---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene Memory Leak

장용석
In reply to this post by Andy33
I think when your doQuery method is run, Directory and Analyzer classes are
new create every time.
If index file's size is very large then create new Directory instance is
pressure to jvm and it takes long time for create new Directory instance.
I suggest that modify the code , Analyzer class and Directory class that
singletone pattern.
(If you can....Search Class,too)


2008/9/3, Andy33 <[hidden email]>:

>
>
> I have a memory leak in my lucene search code. I am able to run a few
> queries
> fine, but I eventually run out of memory. Please note that I do close and
> set to null the ivIndexSearcher object elsewhere. Here is the code I am
> using...
>
>
> private synchronized Hits doQuery(String field, String queryStr, Sort
> sortOrder, String indexDirectory) throws Exception
>    {
>        Directory directory = null;
>
>        try
>        {
>            Analyzer analyzer = new StandardAnalyzer();
>
>            directory = FSDirectory.getDirectory(indexDirectory);
>
>            //search the index
>            ivIndexSearcher = new IndexSearcher(directory);
>
>            QueryParser parser = new QueryParser(field, analyzer);
>            Query query = parser.parse(queryStr);
>            Hits results = ivIndexSearcher.search(query, sortOrder);
>
>            return results;
>        }
>        finally
>        {
>            if(null != directory)
>            {
>                directory.close();
>            }
>            directory = null;
>        }
>    }
> --
> View this message in context:
> http://www.nabble.com/Lucene-Memory-Leak-tp19276999p19276999.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
DEV용식
http://devyongsik.tistory.com
Reply | Threaded
Open this post in threaded view
|

Re: Lucene Memory Leak

Andy33
I took your advice and created Singletons for the Directory, Analyzer, and IndexSearcher classes. I also undid the closing of the Directory and IndexSearcher. This seemed to fix my memory leak problem. However, I don't like the fact that I am leaving open the IndexSearcher for the entire life of a web application. When I close the IndexSearcher, the underlying IndexReader is also closed. It looks like the only way I can reopen the IndexSearcher is to reopen the IndexReader and create a new IndexSearcher. This leads me back to my original problem.

Is there a better way to handle this rather than keeping the IndexSeacher open for the life of the application?


장용석 wrote
I think when your doQuery method is run, Directory and Analyzer classes are
new create every time.
If index file's size is very large then create new Directory instance is
pressure to jvm and it takes long time for create new Directory instance.
I suggest that modify the code , Analyzer class and Directory class that
singletone pattern.
(If you can....Search Class,too)
Reply | Threaded
Open this post in threaded view
|

Re: Lucene Memory Leak

Simon Willnauer
If you are looking for a reasonable performance you should not close
your IndexSearcher if not necessary. It is actually best practice to
leave an IndexSearcher instance open an even share it between threads
/ requests of your webapplication. The searcher will not pollute your
memory. Just keep the searcher open to utilize the lucene internals
like caches and reopen the searcher only if really required this will
most likely give you the best performance.

regards simon

On Wed, Sep 3, 2008 at 7:31 PM, Andy33 <[hidden email]> wrote:

>
> I took your advice and created Singletons for the Directory, Analyzer, and
> IndexSearcher classes. I also undid the closing of the Directory and
> IndexSearcher. This seemed to fix my memory leak problem. However, I don't
> like the fact that I am leaving open the IndexSearcher for the entire life
> of a web application. When I close the IndexSearcher, the underlying
> IndexReader is also closed. It looks like the only way I can reopen the
> IndexSearcher is to reopen the IndexReader and create a new IndexSearcher.
> This leads me back to my original problem.
>
> Is there a better way to handle this rather than keeping the IndexSeacher
> open for the life of the application?
>
>
>
> 장용석 wrote:
>>
>> I think when your doQuery method is run, Directory and Analyzer classes
>> are
>> new create every time.
>> If index file's size is very large then create new Directory instance is
>> pressure to jvm and it takes long time for create new Directory instance.
>> I suggest that modify the code , Analyzer class and Directory class that
>> singletone pattern.
>> (If you can....Search Class,too)
>>
>
> --
> View this message in context: http://www.nabble.com/Lucene-Memory-Leak-tp19276999p19294053.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Lucene Memory Leak

장용석
In reply to this post by Andy33
In fact, I think that the important reasons are Directory class and Analyzer
class.
If you don't want IndexSearcher class keep open for the entire life of a web
application, you can do it.
I think It will not cause memory leak problem.
But, Directory and Analyzer classes can cause the problem if they new
created by method call every time. I think...
Only keep two classes Directory and Analyzer by singlton and do test. :)




2008/9/4, Andy33 <[hidden email]>:

>
>
> I took your advice and created Singletons for the Directory, Analyzer, and
> IndexSearcher classes. I also undid the closing of the Directory and
> IndexSearcher. This seemed to fix my memory leak problem. However, I don't
> like the fact that I am leaving open the IndexSearcher for the entire life
> of a web application. When I close the IndexSearcher, the underlying
> IndexReader is also closed. It looks like the only way I can reopen the
> IndexSearcher is to reopen the IndexReader and create a new IndexSearcher.
> This leads me back to my original problem.
>
> Is there a better way to handle this rather than keeping the IndexSeacher
> open for the life of the application?
>
>
>
> 장용석 wrote:
> >
> > I think when your doQuery method is run, Directory and Analyzer classes
> > are
> > new create every time.
> > If index file's size is very large then create new Directory instance is
> > pressure to jvm and it takes long time for create new Directory instance.
> > I suggest that modify the code , Analyzer class and Directory class that
> > singletone pattern.
> > (If you can....Search Class,too)
> >
>
> --
> View this message in context:
> http://www.nabble.com/Lucene-Memory-Leak-tp19276999p19294053.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
DEV용식
http://devyongsik.tistory.com
Reply | Threaded
Open this post in threaded view
|

Re: Lucene Memory Leak

Andy33
If I don't keep the IndexSearcher as a Singleton and instead open and close a new one each time, I have a large memory leak (probably due to the large queries I am doing). After watching the memory a while, I still believe I have a small memory leak even when the Directory, Analyzer, and IndexSearcher are Singletons. My free memory slowly becomes smaller after each query. Any ideas on what that may be due to?

Here's my updated code...

private synchronized Hits doQuery(String field, String queryStr, Sort sortOrder, String indexDirectory) throws Exception
    {
        Directory directory = null;
        Query query = null;
        QueryParser parser = null;
       
        try
        {
            directory = DirectorySingleton.getInstance(indexDirectory);
            ivIndexSearcher = (IndexSearcher) SearcherSingleton.getInstance(directory);
               
            //search the index
            parser = new QueryParser(field, AnalyzerSingleton.getInstance());
            query = parser.parse(queryStr);
       
            return ivIndexSearcher.search(query, sortOrder);
        }
        finally
        {  
            if(null != directory)
            {
                directory.close();
            }
           
            directory = null;
            parser = null;
            query = null;  
        }
    }


--------------
Example Singleton

public class SearcherSingleton
{
    private static volatile HashMap<Directory, Searcher> cvSearches = new HashMap<Directory, Searcher>();
   
   
    protected SearcherSingleton()
    {
       
    }
   
 
    public static Searcher getInstance(Directory directory) throws IOException
    {
        if(!cvSearches.containsKey(directory))
        {
            synchronized(SearcherSingleton.class)
            {
                if(!cvSearches.containsKey(directory))
                {
                    cvSearches.put(directory, new IndexSearcher(directory));
                }
            }
        }
       
        return cvSearches.get(directory);
    }
   
}


장용석 wrote
In fact, I think that the important reasons are Directory class and Analyzer
class.
If you don't want IndexSearcher class keep open for the entire life of a web
application, you can do it.
I think It will not cause memory leak problem.
But, Directory and Analyzer classes can cause the problem if they new
created by method call every time. I think...
Only keep two classes Directory and Analyzer by singlton and do test. :)
Reply | Threaded
Open this post in threaded view
|

Re: Lucene Memory Leak

chrislusf
Are you using RAMDirectory?

I am actually also dealing with a memory leak. My case is only particular to
RAMDirectory.

http://markmail.org/message/dfgcnnjglne3wynp
However, this RAMDirectory case is not as simple as setting searcher=null,
because I found some reference to RAMDirectory is held by some ThreadLocal
variables.(but I need to reuse thread also).

Here is the reference tree:
org.apache.lucene.store.RAMDirectory
  |- directory of org.apache.lucene.store.RAMFile
      |- file of org.apache.lucene.store.RAMInputStream
          |- base of org.apache.lucene.index.CompoundFileReader$CSIndexInput
              |- input of org.apache.lucene.index.SegmentTermEnum
                  |- value of java.lang.ThreadLocal$ThreadLocalMap$Entry


I am trying to track it down now. If anyone knows about it, please let me
know.

--
Chris Lu
-------------------------
Instant Scalable Full-Text Search On Any Database/Application
site: http://www.dbsight.net
demo: http://search.dbsight.com
Lucene Database Search in 3 minutes:
http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
DBSight customer, a shopping comparison site, (anonymous per request) got
2.6 Million Euro funding!


On Fri, Sep 5, 2008 at 8:43 AM, Andy33 <[hidden email]> wrote:

>
> If I don't keep the IndexSearcher as a Singleton and instead open and close
> a
> new one each time, I have a large memory leak (probably due to the large
> queries I am doing). After watching the memory a while, I still believe I
> have a small memory leak even when the Directory, Analyzer, and
> IndexSearcher are Singletons. My free memory slowly becomes smaller after
> each query. Any ideas on what that may be due to?
>
> Here's my updated code...
>
> private synchronized Hits doQuery(String field, String queryStr, Sort
> sortOrder, String indexDirectory) throws Exception
>    {
>        Directory directory = null;
>         Query query = null;
>        QueryParser parser = null;
>
>        try
>        {
>            directory = DirectorySingleton.getInstance(indexDirectory);
>            ivIndexSearcher = (IndexSearcher)
> SearcherSingleton.getInstance(directory);
>
>            //search the index
>            parser = new QueryParser(field,
> AnalyzerSingleton.getInstance());
>            query = parser.parse(queryStr);
>
>            return ivIndexSearcher.search(query, sortOrder);
>         }
>        finally
>        {
>            if(null != directory)
>            {
>                directory.close();
>            }
>
>            directory = null;
>             parser = null;
>            query = null;
>        }
>    }
>
>
> --------------
> Example Singleton
>
> public class SearcherSingleton
> {
>    private static volatile HashMap<Directory, Searcher> cvSearches = new
> HashMap<Directory, Searcher>();
>
>
>    protected SearcherSingleton()
>    {
>
>    }
>
>
>    public static Searcher getInstance(Directory directory) throws
> IOException
>    {
>        if(!cvSearches.containsKey(directory))
>        {
>            synchronized(SearcherSingleton.class)
>            {
>                if(!cvSearches.containsKey(directory))
>                {
>                    cvSearches.put(directory, new IndexSearcher(directory));
>                }
>            }
>        }
>
>        return cvSearches.get(directory);
>     }
>
> }
>
>
>
> 장용석 wrote:
> >
> > In fact, I think that the important reasons are Directory class and
> > Analyzer
> > class.
> > If you don't want IndexSearcher class keep open for the entire life of a
> > web
> > application, you can do it.
> > I think It will not cause memory leak problem.
> > But, Directory and Analyzer classes can cause the problem if they new
> > created by method call every time. I think...
> > Only keep two classes Directory and Analyzer by singlton and do test. :)
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Lucene-Memory-Leak-tp19276999p19333985.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Lucene Memory Leak

Andy33
No, I am using FSDirectory. Unfortunately, my indexes are over 2 GB in size and I don't have a server that has that much free memory just for the indexes.

If you figure out anything, let me know just in case it helps my case as well. Thanks.


chrislusf wrote
Are you using RAMDirectory?

I am actually also dealing with a memory leak. My case is only particular to
RAMDirectory.

http://markmail.org/message/dfgcnnjglne3wynp
However, this RAMDirectory case is not as simple as setting searcher=null,
because I found some reference to RAMDirectory is held by some ThreadLocal
variables.(but I need to reuse thread also).

Here is the reference tree:
org.apache.lucene.store.RAMDirectory
  |- directory of org.apache.lucene.store.RAMFile
      |- file of org.apache.lucene.store.RAMInputStream
          |- base of org.apache.lucene.index.CompoundFileReader$CSIndexInput
              |- input of org.apache.lucene.index.SegmentTermEnum
                  |- value of java.lang.ThreadLocal$ThreadLocalMap$Entry


I am trying to track it down now. If anyone knows about it, please let me
know.

--
Chris Lu
Reply | Threaded
Open this post in threaded view
|

Re: Lucene Memory Leak

N Hira
In reply to this post by Andy33
I'm not an expert, so please take this with a grain of salt, but if  
you return the Hits object, you are inadvertently "holding on" to  
that IndexSearcher, right?

According to the FAQ (http://wiki.apache.org/lucene-java/ 
ImproveSearchingSpeed), iterating over all Hits will result in  
additional queries.  Further, I don't think that you should call  
Directory.close() because it doesn't really do anything if the  
Searcher is still out there ...

If I was in your place, I would try to iterate over the first N  
(100?) results and return the results back to the caller in some  
wrapper object, leaving the Searcher and the Directory open, at least  
for a test.

If you really need to go against the recommended best practice and  
close the IndexSearcher, then I would strongly suggest some kind of  
scheduler that closes, re-opens, and warms up IndexSearchers on some  
kind of timer.

Good luck!

-h
----------------------------------------------------------------------
Hira, N.R.
Cognocys, Inc.
(773) 251-7453


On 05-Sep-2008, at 10:43 AM, Andy33 wrote:

>
> If I don't keep the IndexSearcher as a Singleton and instead open  
> and close a
> new one each time, I have a large memory leak (probably due to the  
> large
> queries I am doing). After watching the memory a while, I still  
> believe I
> have a small memory leak even when the Directory, Analyzer, and
> IndexSearcher are Singletons. My free memory slowly becomes smaller  
> after
> each query. Any ideas on what that may be due to?
>
> Here's my updated code...
>
> private synchronized Hits doQuery(String field, String queryStr, Sort
> sortOrder, String indexDirectory) throws Exception
>     {
> Directory directory = null;
> Query query = null;
> QueryParser parser = null;
>
> try
> {
>    directory = DirectorySingleton.getInstance(indexDirectory);
>    ivIndexSearcher = (IndexSearcher)
> SearcherSingleton.getInstance(directory);
>
>    //search the index
>    parser = new QueryParser(field, AnalyzerSingleton.getInstance());
>    query = parser.parse(queryStr);
>
>    return ivIndexSearcher.search(query, sortOrder);
> }
> finally
> {
>    if(null != directory)
>    {
> directory.close();
>    }
>
>    directory = null;
>    parser = null;
>    query = null;
> }
>     }
>
>
> --------------
> Example Singleton
>
> public class SearcherSingleton
> {
>     private static volatile HashMap<Directory, Searcher> cvSearches  
> = new
> HashMap<Directory, Searcher>();
>
>
>     protected SearcherSingleton()
>     {
>
>     }
>
>
>     public static Searcher getInstance(Directory directory) throws
> IOException
>     {
> if(!cvSearches.containsKey(directory))
> {
>    synchronized(SearcherSingleton.class)
>    {
> if(!cvSearches.containsKey(directory))
> {
>    cvSearches.put(directory, new IndexSearcher(directory));
> }
>    }
> }
>
> return cvSearches.get(directory);
>     }
>
> }
>
>
>
> 장용석 wrote:
>>
>> In fact, I think that the important reasons are Directory class and
>> Analyzer
>> class.
>> If you don't want IndexSearcher class keep open for the entire  
>> life of a
>> web
>> application, you can do it.
>> I think It will not cause memory leak problem.
>> But, Directory and Analyzer classes can cause the problem if they new
>> created by method call every time. I think...
>> Only keep two classes Directory and Analyzer by singlton and do  
>> test. :)
>>
>>
>





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene Memory Leak

叶双明
In my opinion, do no need to close the Directory, and keep all Directory and
all IndexSearcher open.

return ivIndexSearcher.search(query, sortOrder);  ( I think) is also return
the hits getted frmo IndexSearcher, so it is iterate over the first N, no
problem.

In addition, how much index Directory do you have?  is there a large number
of data in one Document?

2008/9/6, N. Hira <[hidden email]>:

>
> I'm not an expert, so please take this with a grain of salt, but if you
> return the Hits object, you are inadvertently "holding on" to that
> IndexSearcher, right?
>
> According to the FAQ (http://wiki.apache.org/lucene-java/ImproveSearchingSpeed),
> iterating over all Hits will result in additional queries.  Further, I don't
> think that you should call Directory.close() because it doesn't really do
> anything if the Searcher is still out there ...
>
> If I was in your place, I would try to iterate over the first N (100?)
> results and return the results back to the caller in some wrapper object,
> leaving the Searcher and the Directory open, at least for a test.
>
> If you really need to go against the recommended best practice and close
> the IndexSearcher, then I would strongly suggest some kind of scheduler that
> closes, re-opens, and warms up IndexSearchers on some kind of timer.
>
> Good luck!
>
> -h
> ----------------------------------------------------------------------
> Hira, N.R.
> Cognocys, Inc.
> (773) 251-7453
>
>
> On 05-Sep-2008, at 10:43 AM, Andy33 wrote:
>
>
>> If I don't keep the IndexSearcher as a Singleton and instead open and
>> close a
>> new one each time, I have a large memory leak (probably due to the large
>> queries I am doing). After watching the memory a while, I still believe I
>> have a small memory leak even when the Directory, Analyzer, and
>> IndexSearcher are Singletons. My free memory slowly becomes smaller after
>> each query. Any ideas on what that may be due to?
>>
>> Here's my updated code...
>>
>> private synchronized Hits doQuery(String field, String queryStr, Sort
>> sortOrder, String indexDirectory) throws Exception
>>    {
>>        Directory directory = null;
>>        Query query = null;
>>        QueryParser parser = null;
>>
>>        try
>>        {
>>            directory = DirectorySingleton.getInstance(indexDirectory);
>>            ivIndexSearcher = (IndexSearcher)
>> SearcherSingleton.getInstance(directory);
>>
>>            //search the index
>>            parser = new QueryParser(field,
>> AnalyzerSingleton.getInstance());
>>            query = parser.parse(queryStr);
>>
>>            return ivIndexSearcher.search(query, sortOrder);
>>        }
>>        finally
>>        {
>>            if(null != directory)
>>            {
>>                directory.close();
>>            }
>>
>>            directory = null;
>>            parser = null;
>>            query = null;
>>        }
>>    }
>>
>>
>> --------------
>> Example Singleton
>>
>> public class SearcherSingleton
>> {
>>    private static volatile HashMap<Directory, Searcher> cvSearches = new
>> HashMap<Directory, Searcher>();
>>
>>
>>    protected SearcherSingleton()
>>    {
>>
>>    }
>>
>>
>>    public static Searcher getInstance(Directory directory) throws
>> IOException
>>    {
>>        if(!cvSearches.containsKey(directory))
>>        {
>>            synchronized(SearcherSingleton.class)
>>            {
>>                if(!cvSearches.containsKey(directory))
>>                {
>>                    cvSearches.put(directory, new
>> IndexSearcher(directory));
>>                }
>>            }
>>        }
>>
>>        return cvSearches.get(directory);
>>    }
>>
>> }
>>
>>
>>
>> 장용석 wrote:
>>
>>>
>>> In fact, I think that the important reasons are Directory class and
>>> Analyzer
>>> class.
>>> If you don't want IndexSearcher class keep open for the entire life of a
>>> web
>>> application, you can do it.
>>> I think It will not cause memory leak problem.
>>> But, Directory and Analyzer classes can cause the problem if they new
>>> created by method call every time. I think...
>>> Only keep two classes Directory and Analyzer by singlton and do test. :)
>>>
>>>
>>>
>>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Lucene Memory Leak

장용석
In reply to this post by Andy33
Hi.. :)

I saw your code.
Your free memory slowly becomes smaller and start gc? or out of memory
exception?

Declare variable in method use jvm's memory small, and after finish the
method it becomes target of gc.
(example ,
       Directory directory = null;
       Query query = null;
       QueryParser parser = null;
)

That's just a guess..:-)


2008/9/6, Andy33 <[hidden email]>:

>
>
> If I don't keep the IndexSearcher as a Singleton and instead open and close
> a
> new one each time, I have a large memory leak (probably due to the large
> queries I am doing). After watching the memory a while, I still believe I
> have a small memory leak even when the Directory, Analyzer, and
> IndexSearcher are Singletons. My free memory slowly becomes smaller after
> each query. Any ideas on what that may be due to?
>
> Here's my updated code...
>
> private synchronized Hits doQuery(String field, String queryStr, Sort
> sortOrder, String indexDirectory) throws Exception
>    {
>        Directory directory = null;
>        Query query = null;
>        QueryParser parser = null;
>
>        try
>        {
>            directory = DirectorySingleton.getInstance(indexDirectory);
>            ivIndexSearcher = (IndexSearcher)
> SearcherSingleton.getInstance(directory);
>
>            //search the index
>            parser = new QueryParser(field,
> AnalyzerSingleton.getInstance());
>            query = parser.parse(queryStr);
>
>            return ivIndexSearcher.search(query, sortOrder);
>        }
>        finally
>        {
>            if(null != directory)
>            {
>                directory.close();
>            }
>
>            directory = null;
>            parser = null;
>            query = null;
>        }
>    }
>
>
> --------------
> Example Singleton
>
> public class SearcherSingleton
> {
>    private static volatile HashMap<Directory, Searcher> cvSearches = new
> HashMap<Directory, Searcher>();
>
>
>    protected SearcherSingleton()
>    {
>
>    }
>
>
>    public static Searcher getInstance(Directory directory) throws
> IOException
>    {
>        if(!cvSearches.containsKey(directory))
>        {
>            synchronized(SearcherSingleton.class)
>            {
>                if(!cvSearches.containsKey(directory))
>                {
>                    cvSearches.put(directory, new IndexSearcher(directory));
>                }
>            }
>        }
>
>        return cvSearches.get(directory);
>    }
>
> }
>
>
>
> 장용석 wrote:
> >
> > In fact, I think that the important reasons are Directory class and
> > Analyzer
> > class.
> > If you don't want IndexSearcher class keep open for the entire life of a
> > web
> > application, you can do it.
> > I think It will not cause memory leak problem.
> > But, Directory and Analyzer classes can cause the problem if they new
> > created by method call every time. I think...
> > Only keep two classes Directory and Analyzer by singlton and do test. :)
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Lucene-Memory-Leak-tp19276999p19333985.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
DEV용식
http://devyongsik.tistory.com