SolrIndexSearcher accumulation

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

SolrIndexSearcher accumulation

Gerald Reinhart

Hi,

    We have some custom code that extends SearchHandler to be able to :
         - do an extra request
         - merge/combine the original request and the extra request results

    On Solr 5.x, our code was working very well, now with Solr 6.x we
have the following issue:  the number of SolrIndexSearcher are
increasing (we can see them in the admin view > Plugins/ Stats > Core ).
As SolrIndexSearcher are accumulating, we have the following issues :
        - the memory used by Solr is increasing => OOM after a long
period of time in production
        - some files in the index has been deleted from the system but
the Solr JVM still hold them => ("fake") Full disk after a long period
of time in production

    We are wondering,
       - what has changed between Solr 5.x and Solr 6.x in the
management of the SolrIndexSearcher ?
       - what would be the best way, in a Solr plugin, to perform 2
queries and merge the results to a single SolrQueryResponse ?

    Thanks a lot.

Gérald, Elodie, Ludo and André



Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 158 Ter Rue du Temple 75003 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'en avertir l'expéditeur.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SolrIndexSearcher accumulation

Shawn Heisey-2
On 4/7/2017 3:09 AM, Gerald Reinhart wrote:

>    We have some custom code that extends SearchHandler to be able to :
>         - do an extra request
>         - merge/combine the original request and the extra request
> results
>
>    On Solr 5.x, our code was working very well, now with Solr 6.x we
> have the following issue:  the number of SolrIndexSearcher are
> increasing (we can see them in the admin view > Plugins/ Stats > Core ).
> As SolrIndexSearcher are accumulating, we have the following issues :
>        - the memory used by Solr is increasing => OOM after a long
> period of time in production
>        - some files in the index has been deleted from the system but
> the Solr JVM still hold them => ("fake") Full disk after a long period
> of time in production
>
>    We are wondering,
>       - what has changed between Solr 5.x and Solr 6.x in the
> management of the SolrIndexSearcher ?
>       - what would be the best way, in a Solr plugin, to perform 2
> queries and merge the results to a single SolrQueryResponse ?

I hesitated to send a reply because when it comes right down to it, I do
not know a whole lot about deep Solr internals.  I tend to do my work
with the code at a higher level, and don't dive down in the depths all
that often.  I am slowly learning, though.  You may need to wait for a
reply from someone who really knows those internals.

It looks like you and I participated in a discussion last month where
you were facing a similar problem with searchers -- deleted index files
being held open.  How did that turn out?  Seems like if that problem
were solved, it would also solve this problem.

Very likely, the fact that the plugin worked correctly in 5.x was
actually a bug in Solr related to reference counting, one that has been
fixed in later versions.

You may need to use a paste website or a file-sharing website to share
all your plugin code so that people can get a look at it.  The list has
a habit of deleting attachments.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SolrIndexSearcher accumulation

Rick Leir-2
Hi Gerald
The best solution in my mind is to look at the custom code and try to find a way to remove it from your system. Solr queries can be complex, and I hope there is a way to get the results you need. Would you like to say what results you want to get, and what Solr queries you have tried?
I realize that in large organizations it is difficult to suggest change.
Cheers -- Rick

On April 7, 2017 9:08:19 AM EDT, Shawn Heisey <[hidden email]> wrote:

>On 4/7/2017 3:09 AM, Gerald Reinhart wrote:
>>    We have some custom code that extends SearchHandler to be able to
>:
>>         - do an extra request
>>         - merge/combine the original request and the extra request
>> results
>>
>>    On Solr 5.x, our code was working very well, now with Solr 6.x we
>> have the following issue:  the number of SolrIndexSearcher are
>> increasing (we can see them in the admin view > Plugins/ Stats > Core
>).
>> As SolrIndexSearcher are accumulating, we have the following issues :
>>        - the memory used by Solr is increasing => OOM after a long
>> period of time in production
>>        - some files in the index has been deleted from the system but
>> the Solr JVM still hold them => ("fake") Full disk after a long
>period
>> of time in production
>>
>>    We are wondering,
>>       - what has changed between Solr 5.x and Solr 6.x in the
>> management of the SolrIndexSearcher ?
>>       - what would be the best way, in a Solr plugin, to perform 2
>> queries and merge the results to a single SolrQueryResponse ?
>
>I hesitated to send a reply because when it comes right down to it, I
>do
>not know a whole lot about deep Solr internals.  I tend to do my work
>with the code at a higher level, and don't dive down in the depths all
>that often.  I am slowly learning, though.  You may need to wait for a
>reply from someone who really knows those internals.
>
>It looks like you and I participated in a discussion last month where
>you were facing a similar problem with searchers -- deleted index files
>being held open.  How did that turn out?  Seems like if that problem
>were solved, it would also solve this problem.
>
>Very likely, the fact that the plugin worked correctly in 5.x was
>actually a bug in Solr related to reference counting, one that has been
>fixed in later versions.
>
>You may need to use a paste website or a file-sharing website to share
>all your plugin code so that people can get a look at it.  The list has
>a habit of deleting attachments.
>
>Thanks,
>Shawn

--
Sorry for being brief. Alternate email is rickleir at yahoo dot com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SolrIndexSearcher accumulation

Elodie Sannier
Hello,

We have found how to fix the problem.
When we update the original SolrQueryResponse object, we need to create
a new BasicResultContext object with the extra response.

Simplified code :

public class CustomSearchHandler extends SearchHandler {

public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse
rsp) throws Exception {

   SolrQueryRequest extraReq = createExtraRequest();
   SolrQueryResponse extraRsp = new SolrQueryResponse();

   super.handleRequestBody(extraReq, extraRsp);

   ResultContext extraRc = (ResultContext) extraRsp.getResponse();

   // code with memory leak !!
   rsp.addResponse(extraRc);

   // code without memory leak
   ResultContext extraRcClone = new BasicResultContext(extraRc.getDocList(),
             rsp.getReturnFields(), req.getSearcher(),
extraRc.getQuery(), req);
   rsp.addResponse(extraRcClone);

}

}

We don't know why we need to create a new BasicResultContext to properly
manage searchers. Do you know why ?

Elodie

On 04/07/2017 04:14 PM, Rick Leir wrote:

> Hi Gerald
> The best solution in my mind is to look at the custom code and try to find a way to remove it from your system. Solr queries can be complex, and I hope there is a way to get the results you need. Would you like to say what results you want to get, and what Solr queries you have tried?
> I realize that in large organizations it is difficult to suggest change.
> Cheers -- Rick
>
> On April 7, 2017 9:08:19 AM EDT, Shawn Heisey <[hidden email]> wrote:
>> On 4/7/2017 3:09 AM, Gerald Reinhart wrote:
>>>     We have some custom code that extends SearchHandler to be able to
>> :
>>>          - do an extra request
>>>          - merge/combine the original request and the extra request
>>> results
>>>
>>>     On Solr 5.x, our code was working very well, now with Solr 6.x we
>>> have the following issue:  the number of SolrIndexSearcher are
>>> increasing (we can see them in the admin view > Plugins/ Stats > Core
>> ).
>>> As SolrIndexSearcher are accumulating, we have the following issues :
>>>         - the memory used by Solr is increasing => OOM after a long
>>> period of time in production
>>>         - some files in the index has been deleted from the system but
>>> the Solr JVM still hold them => ("fake") Full disk after a long
>> period
>>> of time in production
>>>
>>>     We are wondering,
>>>        - what has changed between Solr 5.x and Solr 6.x in the
>>> management of the SolrIndexSearcher ?
>>>        - what would be the best way, in a Solr plugin, to perform 2
>>> queries and merge the results to a single SolrQueryResponse ?
>> I hesitated to send a reply because when it comes right down to it, I
>> do
>> not know a whole lot about deep Solr internals.  I tend to do my work
>> with the code at a higher level, and don't dive down in the depths all
>> that often.  I am slowly learning, though.  You may need to wait for a
>> reply from someone who really knows those internals.
>>
>> It looks like you and I participated in a discussion last month where
>> you were facing a similar problem with searchers -- deleted index files
>> being held open.  How did that turn out?  Seems like if that problem
>> were solved, it would also solve this problem.
>>
>> Very likely, the fact that the plugin worked correctly in 5.x was
>> actually a bug in Solr related to reference counting, one that has been
>> fixed in later versions.
>>
>> You may need to use a paste website or a file-sharing website to share
>> all your plugin code so that people can get a look at it.  The list has
>> a habit of deleting attachments.
>>
>> Thanks,
>> Shawn


Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 158 Ter Rue du Temple 75003 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'en avertir l'expéditeur.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SolrIndexSearcher accumulation

Mikhail Khludnev-2
If you create SolrQueryRequest make sure you close it then, since it's
necessary to release a searcher.

On Wed, Apr 19, 2017 at 12:35 PM, Elodie Sannier <[hidden email]>
wrote:

> Hello,
>
> We have found how to fix the problem.
> When we update the original SolrQueryResponse object, we need to create
> a new BasicResultContext object with the extra response.
>
> Simplified code :
>
> public class CustomSearchHandler extends SearchHandler {
>
> public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse
> rsp) throws Exception {
>
>   SolrQueryRequest extraReq = createExtraRequest();
>   SolrQueryResponse extraRsp = new SolrQueryResponse();
>
>   super.handleRequestBody(extraReq, extraRsp);
>
>   ResultContext extraRc = (ResultContext) extraRsp.getResponse();
>
>   // code with memory leak !!
>   rsp.addResponse(extraRc);
>
>   // code without memory leak
>   ResultContext extraRcClone = new BasicResultContext(extraRc.get
> DocList(),
>             rsp.getReturnFields(), req.getSearcher(),
> extraRc.getQuery(), req);
>   rsp.addResponse(extraRcClone);
>
> }
>
> }
>
> We don't know why we need to create a new BasicResultContext to properly
> manage searchers. Do you know why ?
>
> Elodie
>
>
> On 04/07/2017 04:14 PM, Rick Leir wrote:
>
>> Hi Gerald
>> The best solution in my mind is to look at the custom code and try to
>> find a way to remove it from your system. Solr queries can be complex, and
>> I hope there is a way to get the results you need. Would you like to say
>> what results you want to get, and what Solr queries you have tried?
>> I realize that in large organizations it is difficult to suggest change.
>> Cheers -- Rick
>>
>> On April 7, 2017 9:08:19 AM EDT, Shawn Heisey <[hidden email]>
>> wrote:
>>
>>> On 4/7/2017 3:09 AM, Gerald Reinhart wrote:
>>>
>>>>     We have some custom code that extends SearchHandler to be able to
>>>>
>>> :
>>>
>>>>          - do an extra request
>>>>          - merge/combine the original request and the extra request
>>>> results
>>>>
>>>>     On Solr 5.x, our code was working very well, now with Solr 6.x we
>>>> have the following issue:  the number of SolrIndexSearcher are
>>>> increasing (we can see them in the admin view > Plugins/ Stats > Core
>>>>
>>> ).
>>>
>>>> As SolrIndexSearcher are accumulating, we have the following issues :
>>>>         - the memory used by Solr is increasing => OOM after a long
>>>> period of time in production
>>>>         - some files in the index has been deleted from the system but
>>>> the Solr JVM still hold them => ("fake") Full disk after a long
>>>>
>>> period
>>>
>>>> of time in production
>>>>
>>>>     We are wondering,
>>>>        - what has changed between Solr 5.x and Solr 6.x in the
>>>> management of the SolrIndexSearcher ?
>>>>        - what would be the best way, in a Solr plugin, to perform 2
>>>> queries and merge the results to a single SolrQueryResponse ?
>>>>
>>> I hesitated to send a reply because when it comes right down to it, I
>>> do
>>> not know a whole lot about deep Solr internals.  I tend to do my work
>>> with the code at a higher level, and don't dive down in the depths all
>>> that often.  I am slowly learning, though.  You may need to wait for a
>>> reply from someone who really knows those internals.
>>>
>>> It looks like you and I participated in a discussion last month where
>>> you were facing a similar problem with searchers -- deleted index files
>>> being held open.  How did that turn out?  Seems like if that problem
>>> were solved, it would also solve this problem.
>>>
>>> Very likely, the fact that the plugin worked correctly in 5.x was
>>> actually a bug in Solr related to reference counting, one that has been
>>> fixed in later versions.
>>>
>>> You may need to use a paste website or a file-sharing website to share
>>> all your plugin code so that people can get a look at it.  The list has
>>> a habit of deleting attachments.
>>>
>>> Thanks,
>>> Shawn
>>>
>>
>
> Kelkoo SAS
> Société par Actions Simplifiée
> Au capital de € 4.168.964,30
> Siège social : 158 Ter Rue du Temple 75003 Paris
> 425 093 069 RCS Paris
>
> Ce message et les pièces jointes sont confidentiels et établis à
> l'attention exclusive de leurs destinataires. Si vous n'êtes pas le
> destinataire de ce message, merci de le détruire et d'en avertir
> l'expéditeur.
>



--
Sincerely yours
Mikhail Khludnev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SolrIndexSearcher accumulation

Elodie Sannier
Yes, I didn't copy all our code but we also do extraReq.close(); in a
finally block. It was not the problem.

On 04/19/2017 11:53 AM, Mikhail Khludnev wrote:

> If you create SolrQueryRequest make sure you close it then, since it's
> necessary to release a searcher.
>
> On Wed, Apr 19, 2017 at 12:35 PM, Elodie Sannier <[hidden email]>
> wrote:
>
>> Hello,
>>
>> We have found how to fix the problem.
>> When we update the original SolrQueryResponse object, we need to create
>> a new BasicResultContext object with the extra response.
>>
>> Simplified code :
>>
>> public class CustomSearchHandler extends SearchHandler {
>>
>> public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse
>> rsp) throws Exception {
>>
>>    SolrQueryRequest extraReq = createExtraRequest();
>>    SolrQueryResponse extraRsp = new SolrQueryResponse();
>>
>>    super.handleRequestBody(extraReq, extraRsp);
>>
>>    ResultContext extraRc = (ResultContext) extraRsp.getResponse();
>>
>>    // code with memory leak !!
>>    rsp.addResponse(extraRc);
>>
>>    // code without memory leak
>>    ResultContext extraRcClone = new BasicResultContext(extraRc.get
>> DocList(),
>>              rsp.getReturnFields(), req.getSearcher(),
>> extraRc.getQuery(), req);
>>    rsp.addResponse(extraRcClone);
>>
>> }
>>
>> }
>>
>> We don't know why we need to create a new BasicResultContext to properly
>> manage searchers. Do you know why ?
>>
>> Elodie
>>
>>
>> On 04/07/2017 04:14 PM, Rick Leir wrote:
>>
>>> Hi Gerald
>>> The best solution in my mind is to look at the custom code and try to
>>> find a way to remove it from your system. Solr queries can be complex, and
>>> I hope there is a way to get the results you need. Would you like to say
>>> what results you want to get, and what Solr queries you have tried?
>>> I realize that in large organizations it is difficult to suggest change.
>>> Cheers -- Rick
>>>
>>> On April 7, 2017 9:08:19 AM EDT, Shawn Heisey <[hidden email]>
>>> wrote:
>>>
>>>> On 4/7/2017 3:09 AM, Gerald Reinhart wrote:
>>>>
>>>>>      We have some custom code that extends SearchHandler to be able to
>>>>>
>>>> :
>>>>
>>>>>           - do an extra request
>>>>>           - merge/combine the original request and the extra request
>>>>> results
>>>>>
>>>>>      On Solr 5.x, our code was working very well, now with Solr 6.x we
>>>>> have the following issue:  the number of SolrIndexSearcher are
>>>>> increasing (we can see them in the admin view > Plugins/ Stats > Core
>>>>>
>>>> ).
>>>>
>>>>> As SolrIndexSearcher are accumulating, we have the following issues :
>>>>>          - the memory used by Solr is increasing => OOM after a long
>>>>> period of time in production
>>>>>          - some files in the index has been deleted from the system but
>>>>> the Solr JVM still hold them => ("fake") Full disk after a long
>>>>>
>>>> period
>>>>
>>>>> of time in production
>>>>>
>>>>>      We are wondering,
>>>>>         - what has changed between Solr 5.x and Solr 6.x in the
>>>>> management of the SolrIndexSearcher ?
>>>>>         - what would be the best way, in a Solr plugin, to perform 2
>>>>> queries and merge the results to a single SolrQueryResponse ?
>>>>>
>>>> I hesitated to send a reply because when it comes right down to it, I
>>>> do
>>>> not know a whole lot about deep Solr internals.  I tend to do my work
>>>> with the code at a higher level, and don't dive down in the depths all
>>>> that often.  I am slowly learning, though.  You may need to wait for a
>>>> reply from someone who really knows those internals.
>>>>
>>>> It looks like you and I participated in a discussion last month where
>>>> you were facing a similar problem with searchers -- deleted index files
>>>> being held open.  How did that turn out?  Seems like if that problem
>>>> were solved, it would also solve this problem.
>>>>
>>>> Very likely, the fact that the plugin worked correctly in 5.x was
>>>> actually a bug in Solr related to reference counting, one that has been
>>>> fixed in later versions.
>>>>
>>>> You may need to use a paste website or a file-sharing website to share
>>>> all your plugin code so that people can get a look at it.  The list has
>>>> a habit of deleting attachments.
>>>>
>>>> Thanks,
>>>> Shawn
>>>>
>> Kelkoo SAS
>> Société par Actions Simplifiée
>> Au capital de € 4.168.964,30
>> Siège social : 158 Ter Rue du Temple 75003 Paris
>> 425 093 069 RCS Paris
>>
>> Ce message et les pièces jointes sont confidentiels et établis à
>> l'attention exclusive de leurs destinataires. Si vous n'êtes pas le
>> destinataire de ce message, merci de le détruire et d'en avertir
>> l'expéditeur.
>>
>
>


Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 158 Ter Rue du Temple 75003 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'en avertir l'expéditeur.
Loading...