SolrIndexSearcher count

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

SolrIndexSearcher count

Rick Dig
Hello all,
is it normal to have many instances (100+) of SolrIndexSearchers to be open
at the same time? Our Heap Analysis shows this to be the case.

We have autoCommit for every 5 minutes, with openSearcher=true, would this
close the old searcher and create a new one or just create a new one with
the old one still not getting dereferenced? if so, when do the older
searchers get cleaned up ?

thanks for your help
-rakshit
Reply | Threaded
Open this post in threaded view
|

Re: SolrIndexSearcher count

Erick Erickson
Do you have any custom code in the mix anywhere?

On Tue, Dec 5, 2017 at 5:02 AM, Rick Dig <[hidden email]> wrote:

> Hello all,
> is it normal to have many instances (100+) of SolrIndexSearchers to be open
> at the same time? Our Heap Analysis shows this to be the case.
>
> We have autoCommit for every 5 minutes, with openSearcher=true, would this
> close the old searcher and create a new one or just create a new one with
> the old one still not getting dereferenced? if so, when do the older
> searchers get cleaned up ?
>
> thanks for your help
> -rakshit
Reply | Threaded
Open this post in threaded view
|

Re: SolrIndexSearcher count

Rick Dig
No custom code at all.

On Dec 5, 2017 10:31 PM, "Erick Erickson" <[hidden email]> wrote:

> Do you have any custom code in the mix anywhere?
>
> On Tue, Dec 5, 2017 at 5:02 AM, Rick Dig <[hidden email]> wrote:
> > Hello all,
> > is it normal to have many instances (100+) of SolrIndexSearchers to be
> open
> > at the same time? Our Heap Analysis shows this to be the case.
> >
> > We have autoCommit for every 5 minutes, with openSearcher=true, would
> this
> > close the old searcher and create a new one or just create a new one with
> > the old one still not getting dereferenced? if so, when do the older
> > searchers get cleaned up ?
> >
> > thanks for your help
> > -rakshit
>
Reply | Threaded
Open this post in threaded view
|

Re: SolrIndexSearcher count

Rick Dig
Hi all
Any pointers on this ?
Thanks


On Dec 5, 2017 22:42, "Rick Dig" <[hidden email]> wrote:

No custom code at all.

On Dec 5, 2017 10:31 PM, "Erick Erickson" <[hidden email]> wrote:

> Do you have any custom code in the mix anywhere?
>
> On Tue, Dec 5, 2017 at 5:02 AM, Rick Dig <[hidden email]> wrote:
> > Hello all,
> > is it normal to have many instances (100+) of SolrIndexSearchers to be
> open
> > at the same time? Our Heap Analysis shows this to be the case.
> >
> > We have autoCommit for every 5 minutes, with openSearcher=true, would
> this
> > close the old searcher and create a new one or just create a new one with
> > the old one still not getting dereferenced? if so, when do the older
> > searchers get cleaned up ?
> >
> > thanks for your help
> > -rakshit
>
Reply | Threaded
Open this post in threaded view
|

Re: SolrIndexSearcher count

Shawn Heisey-2
In reply to this post by Rick Dig
On 12/5/2017 6:02 AM, Rick Dig wrote:
> is it normal to have many instances (100+) of SolrIndexSearchers to be open
> at the same time? Our Heap Analysis shows this to be the case.
>
> We have autoCommit for every 5 minutes, with openSearcher=true, would this
> close the old searcher and create a new one or just create a new one with
> the old one still not getting dereferenced? if so, when do the older
> searchers get cleaned up ?

How many cores is that Solr instance hosting?

Also, have you verified that those searcher objects are actually live
and not slated for garbage collection?  If not, then what I'm saying
below may not apply, and you should find out how many are live before
going any further.  I can imagine a situation where a heap might have
over 100 searcher objects, but most of them are actually dead and ready
for GC.

Each core will have at least one active searcher object at all times. 
When a commit is made that opens a new searcher, you'll temporarily have
an extra searcher object on that core.  The old one SHOULD be removed as
soon as everything using it (which would include queries) is done with
it and the reference counter is fully decremented.  This is why Erick
asked about custom code.  Sometimes when there are Solr plugins that use
the searcher object, the author doesn't close them properly, and that
can lead to the objects building up and never getting removed.

If there are a large number of very slow queries at the same time as
updates are happening and there are frequent commits that open a new
searcher, then you can end up in a situation where there are several
searcher objects which are all running the slow queries at the same time.

I'm not going to claim that a bug in Solr where searchers don't get
closed is impossible, but we haven't seen any evidence in current
versions that this is the case.  Resource leaks have happened in Solr,
but they're very rare.

When I check the list history for the past three years, I only see one
other thread you've been involved in, about a month ago.  The problems
you described at that time and the problem with lots of searcher objects
*might* be related to each other, but I can't be sure.  Can you put a
solr.log file on a sharing/paste website so we can see if there's
anything unusual in it?  You've said that your autoCommit interval is
five minutes, but I suspect that you may have explicit commits happening
on a much more frequent basis.  The solr.log file would reveal what's
actually happening.

Do you have autoSoftCommit configured?  If so, what is the config?  Are
you including a commitWithin parameter on your indexing requests?

Is there ever any time when the Solr instance is mostly idle, when there
are no updates happening and the query rate is zero or very low?  If so,
can you see how many searchers are found on the heap at that time?  Make
sure that the heap dump only includes live objects, so objects that can
be collected as garbage are not included.

Thanks,
Shawn