SolrCore has a large number of SolrIndexSearchers retained in "infoRegistry"

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

SolrCore has a large number of SolrIndexSearchers retained in "infoRegistry"

Jon Poulton
Hi there,
I'm looking at some problems we are having with some legacy code which uses Solr (1.3) under the hood. We seem to get repeated OutOfMemory errors on a 24-48 hour basis searching a relatively small index of 70,000 documents. This may be an error in the way Solr is configured, or it may be a problem with the way it's being invoked, I don't know, as I'm fairly new to Solr.

The OutOfMemoryError is an unusual one:

java.lang.OutOfMemoryError: GC overhead limit exceeded

There are also a few stack traces in the solr logs. The first is:

 SEVERE: Error during auto-warming of key:root_show_stop_range:[2009-12-22T08:59:37.254 TO *]:java.lang.OutOfMemoryError: GC overhead limit exceeded
                at java.util.Arrays.copyOfRange(Arrays.java:3209)
                at java.lang.String.<init>(String.java:215)
                at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122)
                at org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:167)
                at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:251)
                at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:218)
                at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:55)
                at org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.termDocs(MultiSegmentReader.java:609)
                at org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.next(MultiSegmentReader.java:560)
                at org.apache.lucene.search.RangeFilter.getDocIdSet(RangeFilter.java:268)
                at org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.<init>(ConstantScoreQuery.java:116)
                at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:81)
                at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:131)
                at org.apache.lucene.search.Searcher.search(Searcher.java:126)
                at org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:601)
                at org.apache.solr.search.SolrIndexSearcher.getDocSet(SolrIndexSearcher.java:507)
                at org.apache.solr.search.SolrIndexSearcher.cacheDocSet(SolrIndexSearcher.java:482)
                at org.apache.solr.search.SolrIndexSearcher$1.regenerateItem(SolrIndexSearcher.java:224)
                at org.apache.solr.search.LRUCache.warm(LRUCache.java:194)
                at org.apache.solr.search.SolrIndexSearcher.warm(SolrIndexSearcher.java:1518)
                at org.apache.solr.core.SolrCore$3.call(SolrCore.java:1018)
                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
                at java.util.concurrent.FutureTask.run(FutureTask.java:138)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:619)

I took a memory dump of the running application, and found that vast amounts of memory (over 400MB) was being consumed by nine or ten large SolrIndexSearcher objects; references to which are being held within a LinkedHashMap in SolrCore called "infoRegistry".

I had a quick look at the Solr 1.3.0 source code to try and figure out what was going wrong and whether SolrCore was being used incorrectly in our own source. It looks like whenever a new Searcher is created, it "registers" itself with SolrCore, and this registration places a reference to the Searcher in a LinkedHashMap (the "infoRegistry").

What is puzzling me is why so many SolrIndexSearcher objects are being created, and what the conditions are for their creation and removal. The code I can see in our own product does not use SolrIndexSearcher directly, it simply makes calls to "execute" on SolrCore; so I would normally expect that that class would be managing the Searcher life cycle internally.

Does anyone have any idea as to what may be going on here? I have had a look at the solrconfig.xml file, but it does not seem to depart significantly from the defaults provided.

Thanks in advance for any help.

Jon
Reply | Threaded
Open this post in threaded view
|

Re: SolrCore has a large number of SolrIndexSearchers retained in "infoRegistry"

Grant Ingersoll-2

On Dec 23, 2009, at 10:15 AM, Jon Poulton wrote:

> Hi there,
> I'm looking at some problems we are having with some legacy code which uses Solr (1.3) under the hood. We seem to get repeated OutOfMemory errors on a 24-48 hour basis searching a relatively small index of 70,000 documents. This may be an error in the way Solr is configured, or it may be a problem with the way it's being invoked, I don't know, as I'm fairly new to Solr.
>
> The OutOfMemoryError is an unusual one:
>
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> There are also a few stack traces in the solr logs. The first is:
>
> SEVERE: Error during auto-warming of key:root_show_stop_range:[2009-12-22T08:59:37.254 TO *]:java.lang.OutOfMemoryError: GC overhead limit exceeded
>                at java.util.Arrays.copyOfRange(Arrays.java:3209)
>                at java.lang.String.<init>(String.java:215)
>                at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122)
>                at org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:167)
>                at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:251)
>                at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:218)
>                at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:55)
>                at org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.termDocs(MultiSegmentReader.java:609)
>                at org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.next(MultiSegmentReader.java:560)
>                at org.apache.lucene.search.RangeFilter.getDocIdSet(RangeFilter.java:268)
>                at org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.<init>(ConstantScoreQuery.java:116)
>                at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:81)
>                at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:131)
>                at org.apache.lucene.search.Searcher.search(Searcher.java:126)
>                at org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:601)
>                at org.apache.solr.search.SolrIndexSearcher.getDocSet(SolrIndexSearcher.java:507)
>                at org.apache.solr.search.SolrIndexSearcher.cacheDocSet(SolrIndexSearcher.java:482)
>                at org.apache.solr.search.SolrIndexSearcher$1.regenerateItem(SolrIndexSearcher.java:224)
>                at org.apache.solr.search.LRUCache.warm(LRUCache.java:194)
>                at org.apache.solr.search.SolrIndexSearcher.warm(SolrIndexSearcher.java:1518)
>                at org.apache.solr.core.SolrCore$3.call(SolrCore.java:1018)
>                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>                at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>                at java.lang.Thread.run(Thread.java:619)
>
> I took a memory dump of the running application, and found that vast amounts of memory (over 400MB) was being consumed by nine or ten large SolrIndexSearcher objects; references to which are being held within a LinkedHashMap in SolrCore called "infoRegistry".
>
> I had a quick look at the Solr 1.3.0 source code to try and figure out what was going wrong and whether SolrCore was being used incorrectly in our own source. It looks like whenever a new Searcher is created, it "registers" itself with SolrCore, and this registration places a reference to the Searcher in a LinkedHashMap (the "infoRegistry").
>
> What is puzzling me is why so many SolrIndexSearcher objects are being created, and what the conditions are for their creation and removal. The code I can see in our own product does not use SolrIndexSearcher directly, it simply makes calls to "execute" on SolrCore; so I would normally expect that that class would be managing the Searcher life cycle internally.

It sounds like you are either using embedded mode or you have some custom code.  Are you sure you are releasing your resources correctly?

>
> Does anyone have any idea as to what may be going on here? I have had a look at the solrconfig.xml file, but it does not seem to depart significantly from the defaults provided.
>
> Thanks in advance for any help.
>
> Jon

Reply | Threaded
Open this post in threaded view
|

Re: SolrCore has a large number of SolrIndexSearchers retained in "infoRegistry"

skommuri
Hi,

I was wondering if anyone has found any resolution to this email thread?

Thank you,
Siva Kommuri
Reply | Threaded
Open this post in threaded view
|

Re: SolrCore has a large number of SolrIndexSearchers retained in "infoRegistry"

Chris Hostetter-3
:
: I was wondering if anyone has found any resolution to this email thread?

As Grant asked in his reply when this thread was first started (December 2009)...

>> It sounds like you are either using embedded mode or you have some
>> custom code.  Are you sure you are releasing your resources correctly?

...there was no response to his question for clarification.

the problem, given the info we have to work with, definitely seems to be
that the custom code utilizing the SolrCore directly is not releasing the
resources that it is using in every case.

if you are claling hte execute method, that means you have a
SOlrQueryRequest object -- which means you somehow got an instance of
a SolrIndexSearcher (every SOlrQueryRequest has one assocaited with it)
and you are somehow not releasing that SolrIndexSearcher (probably because
you are not calling close() on your SolrQueryRequest)

But it relaly all depends on how you got ahold of that
SOlrQueryRequest/SolrIndexSearcher pair in the first place ... every
method in SolrCore that gives you access to a SolrIndexSearcher is
documented very clearly on how to "release" it when you are done with it
so the ref count can be decremented.


-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: SolrCore has a large number of SolrIndexSearchers retained in "infoRegistry"

kkrugler

On Jul 27, 2010, at 12:21pm, Chris Hostetter wrote:

> :
> : I was wondering if anyone has found any resolution to this email  
> thread?
>
> As Grant asked in his reply when this thread was first started  
> (December 2009)...
>
>>> It sounds like you are either using embedded mode or you have some
>>> custom code.  Are you sure you are releasing your resources  
>>> correctly?
>
> ...there was no response to his question for clarification.
>
> the problem, given the info we have to work with, definitely seems  
> to be
> that the custom code utilizing the SolrCore directly is not  
> releasing the
> resources that it is using in every case.
>
> if you are claling hte execute method, that means you have a
> SOlrQueryRequest object -- which means you somehow got an instance of
> a SolrIndexSearcher (every SOlrQueryRequest has one assocaited with  
> it)
> and you are somehow not releasing that SolrIndexSearcher (probably  
> because
> you are not calling close() on your SolrQueryRequest)

One thing that bit me previously with using APIs in this area of Solr  
is that if you call CoreContainer.getCore(), this increments the open  
count, so you have to balance each getCore() call with a close() call.

The naming here could be better - I think it's common to have an  
expectation that calls to get something don't change any state. Maybe  
openCore()?

-- Ken

> But it relaly all depends on how you got ahold of that
> SOlrQueryRequest/SolrIndexSearcher pair in the first place ... every
> method in SolrCore that gives you access to a SolrIndexSearcher is
> documented very clearly on how to "release" it when you are done  
> with it
> so the ref count can be decremented.
>
>
> -Hoss

--------------------------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c   w e b   m i n i n g




Reply | Threaded
Open this post in threaded view
|

Re: SolrCore has a large number of SolrIndexSearchers retained in "infoRegistry"

skommuri
In reply to this post by Chris Hostetter-3
Thank you very much Hoss for the reply.

I am using the embedded mode (SolrServer). I am not explicitly accessing SolrIndexSearcher. I am explicitly closing the SolrCore after the request has been processed.

Although I did notice that I am using SolrQueryRequest object and is not explicitly getting closed. I will test that one out and will let you know. Thanks again!

Reply | Threaded
Open this post in threaded view
|

Re: SolrCore has a large number of SolrIndexSearchers retained in "infoRegistry"

skommuri
Hi,

It didn't seem like it improved the situation. The same exception stack traces are found.

I have explicitly defined the index readers to be reopened by specifying in the solrconfig.xml

The exception occurs when the remote cores are being searched. I am attaching the exceptions in a text file for reference. solrexceptions.txt

Couple of notes:

1. QueryComponent#process
     Is requesting for a SolrIndexSearcher twice by calling SolrQueryRequest#getSearcher() but is never being closed. I see several instances where getSearcher is being called but is never being properly closed - performing a quick call heirarchy of SolrQueryRequest#getSearcher() and SolrQueryRequest#close() will illustrate this point.

2. It may be the case that this exception was never encountered because typical deployments are not heavily using Distributed Search across multiple Solr Cores and/or it's a small memory leak and so never noticed?
Reply | Threaded
Open this post in threaded view
|

Re: SolrCore has a large number of SolrIndexSearchers retained in "infoRegistry"

Chris Hostetter-3

: 1. QueryComponent#process
:      Is requesting for a SolrIndexSearcher twice by calling
: SolrQueryRequest#getSearcher() but is never being closed. I see several
: instances where getSearcher is being called but is never being properly
: closed - performing a quick call heirarchy of SolrQueryRequest#getSearcher()
: and SolrQueryRequest#close() will illustrate this point.

i believe you missunderstood my point -- i never said code that calls
SolrQueryRequest#getSearcher needs to call SolrQueryRequest#close ...  i
said that SolrIndexSearcher methods are ref counted, and that they those
refrence counts needto be dealt with properly to ensure that the
SOlrIndexSearchers are closed properly -- the fact that you are using
embedded solr, and the fact that you are calling SolrCore.execute means
that you *must* (by defintion) be instnatiating your own instances of
SolrQueryRequest, which means thta you *must* (by definition) be "getting"
a SolrIndexSearcher object from some method that requires you to "release"
that SolrIndexSearcher when you are done.  If you use Solr as a server, it
takes care of calling SolrQueryRequest.close() for you and it takes care
of releasing the SolrIndexSearcher assocaited with that SolrQueryRequest
object.

I realize now that i wasn't explicit in my previous response but there is
absolutely *no* way that anyone can provide any meaninful assistance to
you in understanding these errors w/o you showing us *exactly* how you are
using solr, and what your custom code looksl ike that you have embedded
solr in -- we need to see how you instantiate your SOlrCore, we need to
see all of the code involved when you call the execute method, we need to
see where you get your SolrQueryRequest object from, we need to see where
you get your SolrIndexSearcher from ... all of it.

As i said: these types of resources leaks are entirely dependent on *how*
you use the code ... if you were using Solr as a server, we would know
exactly how the objects are being accessed/released because that code is
in SOlr -- but when you embedded Solr in custom code we have nothing to go
on but what you tell us.



-Hoss