ygc problem on solr 5.5.1

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

ygc problem on solr 5.5.1

胡一博
Hello everyone!


I run a solr cloud on version 5.5.1.
Sometime ,the ygc time would increase from 0.1s to 10+seconds and keep 10+seconds for several hours.Even after I trigger a fullgc ,the ygc still cost 10+seconds.
This happened seldom.
jvm params:(java version "1.7.0_60", Java HotSpot (TM) 64-Bit Server VM(build 24.60-b09, mixed mode))
-XX:+CMSParallelRemarkEnabled
-XX:+CMSScavengeBeforeRemark
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:+CMSFullGCsBeforeCompaction=1
-XX:+CMSInitiatingOccupancyFraction=80
-XX:ConcGCThreads=4
-XX:MaxTenuringThreshold=8
-XX:NewRatio=3
-XX:ParallelGCThreads=4
-XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90
-Xms30g
-Xmx30g
-Xss256k
Reply | Threaded
Open this post in threaded view
|

Re: ygc problem on solr 5.5.1

Samuel Tatipamula
Hi,

There are a couple of things based on your configuration I can suggest.

-XX:ConcGCThreads=4 - try removing this restriction on the threads.

-Xms30g - you should re-consider this param, as 30 GB is huge heap size.
Instead, in SolrCloud, try spawning multiple instances if you have system
resources. I work on SolrCloud where each server handles 10k read rpm, and
found a 12 GB heap to be more than enough. Also, I found 1:2 ratio
(NewRatio=2) to be more optimal than 1:3.

Thanks,
Samuel

On Fri, Nov 10, 2017 at 11:52 AM, 胡一博 <[hidden email]> wrote:

> Hello everyone!
>
>
> I run a solr cloud on version 5.5.1.
> Sometime ,the ygc time would increase from 0.1s to 10+seconds and keep
> 10+seconds for several hours.Even after I trigger a fullgc ,the ygc still
> cost 10+seconds.
> This happened seldom.
> jvm params:(java version "1.7.0_60", Java HotSpot (TM) 64-Bit Server
> VM(build 24.60-b09, mixed mode))
> -XX:+CMSParallelRemarkEnabled
> -XX:+CMSScavengeBeforeRemark
> -XX:+ParallelRefProcEnabled
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseConcMarkSweepGC
> -XX:+UseParNewGC
> -XX:+CMSFullGCsBeforeCompaction=1
> -XX:+CMSInitiatingOccupancyFraction=80
> -XX:ConcGCThreads=4
> -XX:MaxTenuringThreshold=8
> -XX:NewRatio=3
> -XX:ParallelGCThreads=4
> -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90
> -Xms30g
> -Xmx30g
> -Xss256k
Reply | Threaded
Open this post in threaded view
|

Re:Re: ygc problem on solr 5.5.1

胡一博
Thank you for your suggest!


about the NewRatio param. I found some 'promotion failed' in the gc log. It trigger a stw gc instead of cms gc. If i change the NewRatio to 2,the promotion fail maybe appeard more frequenly. Is the 'promotion failed' is caused by some inappropriate use on solr cloud? .

















At 2017-11-10 16:46:22, "Samuel Tatipamula" <[hidden email]> wrote:

>Hi,
>
>There are a couple of things based on your configuration I can suggest.
>
>-XX:ConcGCThreads=4 - try removing this restriction on the threads.
>
>-Xms30g - you should re-consider this param, as 30 GB is huge heap size.
>Instead, in SolrCloud, try spawning multiple instances if you have system
>resources. I work on SolrCloud where each server handles 10k read rpm, and
>found a 12 GB heap to be more than enough. Also, I found 1:2 ratio
>(NewRatio=2) to be more optimal than 1:3.
>
>Thanks,
>Samuel
>
>On Fri, Nov 10, 2017 at 11:52 AM, 胡一博 <[hidden email]> wrote:
>
>> Hello everyone!
>>
>>
>> I run a solr cloud on version 5.5.1.
>> Sometime ,the ygc time would increase from 0.1s to 10+seconds and keep
>> 10+seconds for several hours.Even after I trigger a fullgc ,the ygc still
>> cost 10+seconds.
>> This happened seldom.
>> jvm params:(java version "1.7.0_60", Java HotSpot (TM) 64-Bit Server
>> VM(build 24.60-b09, mixed mode))
>> -XX:+CMSParallelRemarkEnabled
>> -XX:+CMSScavengeBeforeRemark
>> -XX:+ParallelRefProcEnabled
>> -XX:+UseCMSInitiatingOccupancyOnly
>> -XX:+UseConcMarkSweepGC
>> -XX:+UseParNewGC
>> -XX:+CMSFullGCsBeforeCompaction=1
>> -XX:+CMSInitiatingOccupancyFraction=80
>> -XX:ConcGCThreads=4
>> -XX:MaxTenuringThreshold=8
>> -XX:NewRatio=3
>> -XX:ParallelGCThreads=4
>> -XX:SurvivorRatio=4
>> -XX:TargetSurvivorRatio=90
>> -Xms30g
>> -Xmx30g
>> -Xss256k
Reply | Threaded
Open this post in threaded view
|

Re: Re: ygc problem on solr 5.5.1

Samuel Tatipamula
Promotion failure usually means your old gen doesn't have enough space to
accommodate the incoming (promoted from young to old) object. In your case,
you have specified NewRatio as 3, which means, you have approximately
30*(3/4) = 22.5 GB old gen heap space. If a heap this big gets fragmented,
it would definitely trigger a stw GC to defragment it.

If you change NewRatio to 2, then heap size for old gen will reduce to 20
GB, but still it is huge.

I would recommend decreasing the heap size to a reasonable value (say 12?),
and apply NewRatio of 2, which I found to be working good for my use case.

Also, you have given CMSInitiatingOccupancyFraction as 80, which might be a
bit too late to start triggering CMS. I suggest a value b/w 70-75 for
bigger heaps.

Thanks.

On Fri, Nov 10, 2017 at 3:39 PM, 胡一博 <[hidden email]> wrote:

> Thank you for your suggest!
>
>
> about the NewRatio param. I found some 'promotion failed' in the gc log.
> It trigger a stw gc instead of cms gc. If i change the NewRatio to 2,the
> promotion fail maybe appeard more frequenly. Is the 'promotion failed' is
> caused by some inappropriate use on solr cloud? .
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> At 2017-11-10 16:46:22, "Samuel Tatipamula" <[hidden email]>
> wrote:
> >Hi,
> >
> >There are a couple of things based on your configuration I can suggest.
> >
> >-XX:ConcGCThreads=4 - try removing this restriction on the threads.
> >
> >-Xms30g - you should re-consider this param, as 30 GB is huge heap size.
> >Instead, in SolrCloud, try spawning multiple instances if you have system
> >resources. I work on SolrCloud where each server handles 10k read rpm, and
> >found a 12 GB heap to be more than enough. Also, I found 1:2 ratio
> >(NewRatio=2) to be more optimal than 1:3.
> >
> >Thanks,
> >Samuel
> >
> >On Fri, Nov 10, 2017 at 11:52 AM, 胡一博 <[hidden email]> wrote:
> >
> >> Hello everyone!
> >>
> >>
> >> I run a solr cloud on version 5.5.1.
> >> Sometime ,the ygc time would increase from 0.1s to 10+seconds and keep
> >> 10+seconds for several hours.Even after I trigger a fullgc ,the ygc
> still
> >> cost 10+seconds.
> >> This happened seldom.
> >> jvm params:(java version "1.7.0_60", Java HotSpot (TM) 64-Bit Server
> >> VM(build 24.60-b09, mixed mode))
> >> -XX:+CMSParallelRemarkEnabled
> >> -XX:+CMSScavengeBeforeRemark
> >> -XX:+ParallelRefProcEnabled
> >> -XX:+UseCMSInitiatingOccupancyOnly
> >> -XX:+UseConcMarkSweepGC
> >> -XX:+UseParNewGC
> >> -XX:+CMSFullGCsBeforeCompaction=1
> >> -XX:+CMSInitiatingOccupancyFraction=80
> >> -XX:ConcGCThreads=4
> >> -XX:MaxTenuringThreshold=8
> >> -XX:NewRatio=3
> >> -XX:ParallelGCThreads=4
> >> -XX:SurvivorRatio=4
> >> -XX:TargetSurvivorRatio=90
> >> -Xms30g
> >> -Xmx30g
> >> -Xss256k
>