Solr memory leak

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr memory leak

Hendrik Haddorp
Hi,

we noticed that triggering collection reloads on many collections has a
good chance to result in an OOM-Error. To investigate that further I did
a simple test:
     - Start solr with a 2GB heap and 1GB Metaspace
     - create a trivial collection with a few documents (I used only 2
fields and 100 documents)
     - trigger a collection reload in a loop (I used SolrJ for this)

Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6
worked better but also failed after 1100 loops.

When looking at the memory usage on the Solr dashboard it looks like the
space left after GC cycles gets less and less. Then Solr gets very slow,
as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
my last run this was actually for the Metaspace. So it looks like more
and more heap and metaspace is being used by just constantly reloading a
trivial collection.

regards,
Hendrik
Reply | Threaded
Open this post in threaded view
|

RE: Solr memory leak

Markus Jelsma-2
See https://issues.apache.org/jira/browse/SOLR-10506
Fixed for 7.0

Markus

 
 
-----Original message-----

> From:Hendrik Haddorp <[hidden email]>
> Sent: Monday 28th August 2017 17:42
> To: [hidden email]
> Subject: Solr memory leak
>
> Hi,
>
> we noticed that triggering collection reloads on many collections has a
> good chance to result in an OOM-Error. To investigate that further I did
> a simple test:
>      - Start solr with a 2GB heap and 1GB Metaspace
>      - create a trivial collection with a few documents (I used only 2
> fields and 100 documents)
>      - trigger a collection reload in a loop (I used SolrJ for this)
>
> Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6
> worked better but also failed after 1100 loops.
>
> When looking at the memory usage on the Solr dashboard it looks like the
> space left after GC cycles gets less and less. Then Solr gets very slow,
> as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
> my last run this was actually for the Metaspace. So it looks like more
> and more heap and metaspace is being used by just constantly reloading a
> trivial collection.
>
> regards,
> Hendrik
>
Reply | Threaded
Open this post in threaded view
|

RE: Solr memory leak

Markus Jelsma-2
It is, unfortunately, not committed for 6.7.



 
 
-----Original message-----

> From:Markus Jelsma <[hidden email]>
> Sent: Monday 28th August 2017 17:46
> To: [hidden email]
> Subject: RE: Solr memory leak
>
> See https://issues.apache.org/jira/browse/SOLR-10506
> Fixed for 7.0
>
> Markus
>
>  
>  
> -----Original message-----
> > From:Hendrik Haddorp <[hidden email]>
> > Sent: Monday 28th August 2017 17:42
> > To: [hidden email]
> > Subject: Solr memory leak
> >
> > Hi,
> >
> > we noticed that triggering collection reloads on many collections has a
> > good chance to result in an OOM-Error. To investigate that further I did
> > a simple test:
> >      - Start solr with a 2GB heap and 1GB Metaspace
> >      - create a trivial collection with a few documents (I used only 2
> > fields and 100 documents)
> >      - trigger a collection reload in a loop (I used SolrJ for this)
> >
> > Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6
> > worked better but also failed after 1100 loops.
> >
> > When looking at the memory usage on the Solr dashboard it looks like the
> > space left after GC cycles gets less and less. Then Solr gets very slow,
> > as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
> > my last run this was actually for the Metaspace. So it looks like more
> > and more heap and metaspace is being used by just constantly reloading a
> > trivial collection.
> >
> > regards,
> > Hendrik
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr memory leak

Walter Underwood
That would be a really good reason for a 6.7.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)


> On Aug 28, 2017, at 8:48 AM, Markus Jelsma <[hidden email]> wrote:
>
> It is, unfortunately, not committed for 6.7.
>
>
>
>
>
> -----Original message-----
>> From:Markus Jelsma <[hidden email]>
>> Sent: Monday 28th August 2017 17:46
>> To: [hidden email]
>> Subject: RE: Solr memory leak
>>
>> See https://issues.apache.org/jira/browse/SOLR-10506
>> Fixed for 7.0
>>
>> Markus
>>
>>
>>
>> -----Original message-----
>>> From:Hendrik Haddorp <[hidden email]>
>>> Sent: Monday 28th August 2017 17:42
>>> To: [hidden email]
>>> Subject: Solr memory leak
>>>
>>> Hi,
>>>
>>> we noticed that triggering collection reloads on many collections has a
>>> good chance to result in an OOM-Error. To investigate that further I did
>>> a simple test:
>>>     - Start solr with a 2GB heap and 1GB Metaspace
>>>     - create a trivial collection with a few documents (I used only 2
>>> fields and 100 documents)
>>>     - trigger a collection reload in a loop (I used SolrJ for this)
>>>
>>> Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6
>>> worked better but also failed after 1100 loops.
>>>
>>> When looking at the memory usage on the Solr dashboard it looks like the
>>> space left after GC cycles gets less and less. Then Solr gets very slow,
>>> as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
>>> my last run this was actually for the Metaspace. So it looks like more
>>> and more heap and metaspace is being used by just constantly reloading a
>>> trivial collection.
>>>
>>> regards,
>>> Hendrik
>>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Solr memory leak

Erick Erickson
Varun Thacker is the RM for Solr 6.6.1, I've pinged him about including it.

On Mon, Aug 28, 2017 at 8:52 AM, Walter Underwood <[hidden email]> wrote:

> That would be a really good reason for a 6.7.
>
> wunder
> Walter Underwood
> [hidden email]
> http://observer.wunderwood.org/  (my blog)
>
>
>> On Aug 28, 2017, at 8:48 AM, Markus Jelsma <[hidden email]> wrote:
>>
>> It is, unfortunately, not committed for 6.7.
>>
>>
>>
>>
>>
>> -----Original message-----
>>> From:Markus Jelsma <[hidden email]>
>>> Sent: Monday 28th August 2017 17:46
>>> To: [hidden email]
>>> Subject: RE: Solr memory leak
>>>
>>> See https://issues.apache.org/jira/browse/SOLR-10506
>>> Fixed for 7.0
>>>
>>> Markus
>>>
>>>
>>>
>>> -----Original message-----
>>>> From:Hendrik Haddorp <[hidden email]>
>>>> Sent: Monday 28th August 2017 17:42
>>>> To: [hidden email]
>>>> Subject: Solr memory leak
>>>>
>>>> Hi,
>>>>
>>>> we noticed that triggering collection reloads on many collections has a
>>>> good chance to result in an OOM-Error. To investigate that further I did
>>>> a simple test:
>>>>     - Start solr with a 2GB heap and 1GB Metaspace
>>>>     - create a trivial collection with a few documents (I used only 2
>>>> fields and 100 documents)
>>>>     - trigger a collection reload in a loop (I used SolrJ for this)
>>>>
>>>> Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6
>>>> worked better but also failed after 1100 loops.
>>>>
>>>> When looking at the memory usage on the Solr dashboard it looks like the
>>>> space left after GC cycles gets less and less. Then Solr gets very slow,
>>>> as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
>>>> my last run this was actually for the Metaspace. So it looks like more
>>>> and more heap and metaspace is being used by just constantly reloading a
>>>> trivial collection.
>>>>
>>>> regards,
>>>> Hendrik
>>>>
>>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr memory leak

Hendrik Haddorp
Did you get an answer? Would really be nice to have that in the next
release.

On 28.08.2017 18:31, Erick Erickson wrote:

> Varun Thacker is the RM for Solr 6.6.1, I've pinged him about including it.
>
> On Mon, Aug 28, 2017 at 8:52 AM, Walter Underwood <[hidden email]> wrote:
>> That would be a really good reason for a 6.7.
>>
>> wunder
>> Walter Underwood
>> [hidden email]
>> http://observer.wunderwood.org/  (my blog)
>>
>>
>>> On Aug 28, 2017, at 8:48 AM, Markus Jelsma <[hidden email]> wrote:
>>>
>>> It is, unfortunately, not committed for 6.7.
>>>
>>>
>>>
>>>
>>>
>>> -----Original message-----
>>>> From:Markus Jelsma <[hidden email]>
>>>> Sent: Monday 28th August 2017 17:46
>>>> To: [hidden email]
>>>> Subject: RE: Solr memory leak
>>>>
>>>> See https://issues.apache.org/jira/browse/SOLR-10506
>>>> Fixed for 7.0
>>>>
>>>> Markus
>>>>
>>>>
>>>>
>>>> -----Original message-----
>>>>> From:Hendrik Haddorp <[hidden email]>
>>>>> Sent: Monday 28th August 2017 17:42
>>>>> To: [hidden email]
>>>>> Subject: Solr memory leak
>>>>>
>>>>> Hi,
>>>>>
>>>>> we noticed that triggering collection reloads on many collections has a
>>>>> good chance to result in an OOM-Error. To investigate that further I did
>>>>> a simple test:
>>>>>      - Start solr with a 2GB heap and 1GB Metaspace
>>>>>      - create a trivial collection with a few documents (I used only 2
>>>>> fields and 100 documents)
>>>>>      - trigger a collection reload in a loop (I used SolrJ for this)
>>>>>
>>>>> Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6
>>>>> worked better but also failed after 1100 loops.
>>>>>
>>>>> When looking at the memory usage on the Solr dashboard it looks like the
>>>>> space left after GC cycles gets less and less. Then Solr gets very slow,
>>>>> as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
>>>>> my last run this was actually for the Metaspace. So it looks like more
>>>>> and more heap and metaspace is being used by just constantly reloading a
>>>>> trivial collection.
>>>>>
>>>>> regards,
>>>>> Hendrik
>>>>>

Reply | Threaded
Open this post in threaded view
|

Re: Solr memory leak

Hendrik Haddorp
In reply to this post by Erick Erickson
Hi,

looks like SOLR-10506 didn't make it into 6.6.1. I do however also not
see it listen in the current release notes for 6.7 nor 7.0:
     https://issues.apache.org/jira/projects/SOLR/versions/12340568
     https://issues.apache.org/jira/projects/SOLR/versions/12335718

Is there any any rough idea already when 6.7 or 7.0 will be released?

thanks,
Hendrik

On 28.08.2017 18:31, Erick Erickson wrote:

> Varun Thacker is the RM for Solr 6.6.1, I've pinged him about including it.
>
> On Mon, Aug 28, 2017 at 8:52 AM, Walter Underwood <[hidden email]> wrote:
>> That would be a really good reason for a 6.7.
>>
>> wunder
>> Walter Underwood
>> [hidden email]
>> http://observer.wunderwood.org/  (my blog)
>>
>>
>>> On Aug 28, 2017, at 8:48 AM, Markus Jelsma <[hidden email]> wrote:
>>>
>>> It is, unfortunately, not committed for 6.7.
>>>
>>>
>>>
>>>
>>>
>>> -----Original message-----
>>>> From:Markus Jelsma <[hidden email]>
>>>> Sent: Monday 28th August 2017 17:46
>>>> To: [hidden email]
>>>> Subject: RE: Solr memory leak
>>>>
>>>> See https://issues.apache.org/jira/browse/SOLR-10506
>>>> Fixed for 7.0
>>>>
>>>> Markus
>>>>
>>>>
>>>>
>>>> -----Original message-----
>>>>> From:Hendrik Haddorp <[hidden email]>
>>>>> Sent: Monday 28th August 2017 17:42
>>>>> To: [hidden email]
>>>>> Subject: Solr memory leak
>>>>>
>>>>> Hi,
>>>>>
>>>>> we noticed that triggering collection reloads on many collections has a
>>>>> good chance to result in an OOM-Error. To investigate that further I did
>>>>> a simple test:
>>>>>      - Start solr with a 2GB heap and 1GB Metaspace
>>>>>      - create a trivial collection with a few documents (I used only 2
>>>>> fields and 100 documents)
>>>>>      - trigger a collection reload in a loop (I used SolrJ for this)
>>>>>
>>>>> Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6
>>>>> worked better but also failed after 1100 loops.
>>>>>
>>>>> When looking at the memory usage on the Solr dashboard it looks like the
>>>>> space left after GC cycles gets less and less. Then Solr gets very slow,
>>>>> as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
>>>>> my last run this was actually for the Metaspace. So it looks like more
>>>>> and more heap and metaspace is being used by just constantly reloading a
>>>>> trivial collection.
>>>>>
>>>>> regards,
>>>>> Hendrik
>>>>>

Reply | Threaded
Open this post in threaded view
|

Re: Solr memory leak

Erick Erickson
There will be no 6.7. Once the X+1 version is released, all past fixes
are applied to as minor releases to the last released version of the
previous major release. So now that 7.0 has been cut, there might be a
6.6.2 (6.6.1 was just released) but no 6.7. Current un-released JIRAs
are parked on the 6.x (as opposed to branch_6_6) for convenience. If
anyone steps up to release 6.6.2, they can include ths.

Why do you say this isn't in 7.0? The "Fix Versions" clearly states
so, as does CHANGES.txt for 7.0. The new file is is in the 7.0 branch.


If you need it in 6x you have a couple of options:

1> agitate fo ra 6.6.2 with this included
2> apply the patch yourself and compile it locally

Best,
Erick

On Sun, Sep 10, 2017 at 6:04 AM, Hendrik Haddorp
<[hidden email]> wrote:

> Hi,
>
> looks like SOLR-10506 didn't make it into 6.6.1. I do however also not see
> it listen in the current release notes for 6.7 nor 7.0:
>     https://issues.apache.org/jira/projects/SOLR/versions/12340568
>     https://issues.apache.org/jira/projects/SOLR/versions/12335718
>
> Is there any any rough idea already when 6.7 or 7.0 will be released?
>
> thanks,
> Hendrik
>
>
> On 28.08.2017 18:31, Erick Erickson wrote:
>>
>> Varun Thacker is the RM for Solr 6.6.1, I've pinged him about including
>> it.
>>
>> On Mon, Aug 28, 2017 at 8:52 AM, Walter Underwood <[hidden email]>
>> wrote:
>>>
>>> That would be a really good reason for a 6.7.
>>>
>>> wunder
>>> Walter Underwood
>>> [hidden email]
>>> http://observer.wunderwood.org/  (my blog)
>>>
>>>
>>>> On Aug 28, 2017, at 8:48 AM, Markus Jelsma <[hidden email]>
>>>> wrote:
>>>>
>>>> It is, unfortunately, not committed for 6.7.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -----Original message-----
>>>>>
>>>>> From:Markus Jelsma <[hidden email]>
>>>>> Sent: Monday 28th August 2017 17:46
>>>>> To: [hidden email]
>>>>> Subject: RE: Solr memory leak
>>>>>
>>>>> See https://issues.apache.org/jira/browse/SOLR-10506
>>>>> Fixed for 7.0
>>>>>
>>>>> Markus
>>>>>
>>>>>
>>>>>
>>>>> -----Original message-----
>>>>>>
>>>>>> From:Hendrik Haddorp <[hidden email]>
>>>>>> Sent: Monday 28th August 2017 17:42
>>>>>> To: [hidden email]
>>>>>> Subject: Solr memory leak
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> we noticed that triggering collection reloads on many collections has
>>>>>> a
>>>>>> good chance to result in an OOM-Error. To investigate that further I
>>>>>> did
>>>>>> a simple test:
>>>>>>      - Start solr with a 2GB heap and 1GB Metaspace
>>>>>>      - create a trivial collection with a few documents (I used only 2
>>>>>> fields and 100 documents)
>>>>>>      - trigger a collection reload in a loop (I used SolrJ for this)
>>>>>>
>>>>>> Using Solr 6.3 the test started to fail after about 250 loops. Solr
>>>>>> 6.6
>>>>>> worked better but also failed after 1100 loops.
>>>>>>
>>>>>> When looking at the memory usage on the Solr dashboard it looks like
>>>>>> the
>>>>>> space left after GC cycles gets less and less. Then Solr gets very
>>>>>> slow,
>>>>>> as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
>>>>>> my last run this was actually for the Metaspace. So it looks like more
>>>>>> and more heap and metaspace is being used by just constantly reloading
>>>>>> a
>>>>>> trivial collection.
>>>>>>
>>>>>> regards,
>>>>>> Hendrik
>>>>>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr memory leak

Hendrik Haddorp
I didn't meant to say that the fix is not in 7.0. I just stated that I
do not see it listed in the release notes
(https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310230&version=12335718).

Thanks for explaining the release process.

regards,
Hendrik

On 10.09.2017 17:32, Erick Erickson wrote:

> There will be no 6.7. Once the X+1 version is released, all past fixes
> are applied to as minor releases to the last released version of the
> previous major release. So now that 7.0 has been cut, there might be a
> 6.6.2 (6.6.1 was just released) but no 6.7. Current un-released JIRAs
> are parked on the 6.x (as opposed to branch_6_6) for convenience. If
> anyone steps up to release 6.6.2, they can include ths.
>
> Why do you say this isn't in 7.0? The "Fix Versions" clearly states
> so, as does CHANGES.txt for 7.0. The new file is is in the 7.0 branch.
>
>
> If you need it in 6x you have a couple of options:
>
> 1> agitate fo ra 6.6.2 with this included
> 2> apply the patch yourself and compile it locally
>
> Best,
> Erick
>
> On Sun, Sep 10, 2017 at 6:04 AM, Hendrik Haddorp
> <[hidden email]> wrote:
>> Hi,
>>
>> looks like SOLR-10506 didn't make it into 6.6.1. I do however also not see
>> it listen in the current release notes for 6.7 nor 7.0:
>>      https://issues.apache.org/jira/projects/SOLR/versions/12340568
>>      https://issues.apache.org/jira/projects/SOLR/versions/12335718
>>
>> Is there any any rough idea already when 6.7 or 7.0 will be released?
>>
>> thanks,
>> Hendrik
>>
>>
>> On 28.08.2017 18:31, Erick Erickson wrote:
>>> Varun Thacker is the RM for Solr 6.6.1, I've pinged him about including
>>> it.
>>>
>>> On Mon, Aug 28, 2017 at 8:52 AM, Walter Underwood <[hidden email]>
>>> wrote:
>>>> That would be a really good reason for a 6.7.
>>>>
>>>> wunder
>>>> Walter Underwood
>>>> [hidden email]
>>>> http://observer.wunderwood.org/  (my blog)
>>>>
>>>>
>>>>> On Aug 28, 2017, at 8:48 AM, Markus Jelsma <[hidden email]>
>>>>> wrote:
>>>>>
>>>>> It is, unfortunately, not committed for 6.7.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original message-----
>>>>>> From:Markus Jelsma <[hidden email]>
>>>>>> Sent: Monday 28th August 2017 17:46
>>>>>> To: [hidden email]
>>>>>> Subject: RE: Solr memory leak
>>>>>>
>>>>>> See https://issues.apache.org/jira/browse/SOLR-10506
>>>>>> Fixed for 7.0
>>>>>>
>>>>>> Markus
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original message-----
>>>>>>> From:Hendrik Haddorp <[hidden email]>
>>>>>>> Sent: Monday 28th August 2017 17:42
>>>>>>> To: [hidden email]
>>>>>>> Subject: Solr memory leak
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> we noticed that triggering collection reloads on many collections has
>>>>>>> a
>>>>>>> good chance to result in an OOM-Error. To investigate that further I
>>>>>>> did
>>>>>>> a simple test:
>>>>>>>       - Start solr with a 2GB heap and 1GB Metaspace
>>>>>>>       - create a trivial collection with a few documents (I used only 2
>>>>>>> fields and 100 documents)
>>>>>>>       - trigger a collection reload in a loop (I used SolrJ for this)
>>>>>>>
>>>>>>> Using Solr 6.3 the test started to fail after about 250 loops. Solr
>>>>>>> 6.6
>>>>>>> worked better but also failed after 1100 loops.
>>>>>>>
>>>>>>> When looking at the memory usage on the Solr dashboard it looks like
>>>>>>> the
>>>>>>> space left after GC cycles gets less and less. Then Solr gets very
>>>>>>> slow,
>>>>>>> as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In
>>>>>>> my last run this was actually for the Metaspace. So it looks like more
>>>>>>> and more heap and metaspace is being used by just constantly reloading
>>>>>>> a
>>>>>>> trivial collection.
>>>>>>>
>>>>>>> regards,
>>>>>>> Hendrik
>>>>>>>