Quantcast

Garbage Collection tuning - G1 is now a good option

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Garbage Collection tuning - G1 is now a good option

Shawn Heisey-2
I've been working with Oracle employees to find better GC tuning
options.  The results are good enough to share with the community:

https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning

With the latest Java 7 or Java 8 version, and a couple of tuning
options, G1GC has grown up enough to be a viable choice.  Two of the
settings on that list were critical for making the performance
acceptable with my testing: ParallelRefProcEnabled and G1HeapRegionSize.

I've included some notes on the wiki about how you can size the G1 heap
regions appropriately for your own index.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Garbage Collection tuning - G1 is now a good option

Billnbell
But tons of people on this mailing list do not recommend AggressiveOpts

Why do you recommend it?

On Thu, Jan 1, 2015 at 12:10 PM, Shawn Heisey <[hidden email]> wrote:

> I've been working with Oracle employees to find better GC tuning
> options.  The results are good enough to share with the community:
>
> https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning
>
> With the latest Java 7 or Java 8 version, and a couple of tuning
> options, G1GC has grown up enough to be a viable choice.  Two of the
> settings on that list were critical for making the performance
> acceptable with my testing: ParallelRefProcEnabled and G1HeapRegionSize.
>
> I've included some notes on the wiki about how you can size the G1 heap
> regions appropriately for your own index.
>
> Thanks,
> Shawn
>
>


--
Bill Bell
[hidden email]
cell 720-256-8076
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Garbage Collection tuning - G1 is now a good option

Shawn Heisey-2
On 1/1/2015 6:35 PM, William Bell wrote:
> But tons of people on this mailing list do not recommend AggressiveOpts
>
> Why do you recommend it?

I haven't done any comparisons with and without it.  To call it a
"recommendation" is a little bit strong.  I use it, and I am seeing good
results.

My reading indicates that AggressiveOpts basically enables settings that
are being considered for defaults in a later Java version.  If they are
being seriously considered for new defaults, they are probably something
that I want to be using.  I have also heard that there are sometimes
bugs with that option, but I have not personally had any problems.

If you don't want to use that option, feel free to leave it out.  I will
update my wiki page with a note about AggressiveOpts.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Garbage Collection tuning - G1 is now a good option

Mark Miller-3
bq. But tons of people on this mailing list do not recommend AggressiveOpts

It's up to you to decide - that is why it's an option. It will enable more
aggressive options that will tend to perform better. On the other hand,
these more aggressive options and optimizations have a history of being
more buggy. Depending on your needs and risk tolerance, you might make a
different choice.

You probably want to research the specific issues it has been found to
cause and whether those issues are important to you or affect the JVM you
are using. You also have to keep in mind that probably all of the issues
are not known or documented and that updates may introduce new issues. You
face some risk like that no matter what though.

- Mark

On Fri Jan 02 2015 at 8:37:47 AM Shawn Heisey <[hidden email]> wrote:

> On 1/1/2015 6:35 PM, William Bell wrote:
> > But tons of people on this mailing list do not recommend AggressiveOpts
> >
> > Why do you recommend it?
>
> I haven't done any comparisons with and without it.  To call it a
> "recommendation" is a little bit strong.  I use it, and I am seeing good
> results.
>
> My reading indicates that AggressiveOpts basically enables settings that
> are being considered for defaults in a later Java version.  If they are
> being seriously considered for new defaults, they are probably something
> that I want to be using.  I have also heard that there are sometimes
> bugs with that option, but I have not personally had any problems.
>
> If you don't want to use that option, feel free to leave it out.  I will
> update my wiki page with a note about AggressiveOpts.
>
> Thanks,
> Shawn
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Garbage Collection tuning - G1 is now a good option

Otis Gospodnetić
In reply to this post by Billnbell
Not sure about AggressiveOpts, but G1 has been working for us nicely.
We've successfully used it with HBase, Hadoop, Elasticsearch, and other
custom Java apps (all still Java 7, but Java 8 should be even better).  Not
sure if we are using in on our Solr instances.

e.g. see http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Thu, Jan 1, 2015 at 8:35 PM, William Bell <[hidden email]> wrote:

> But tons of people on this mailing list do not recommend AggressiveOpts
>
> Why do you recommend it?
>
> On Thu, Jan 1, 2015 at 12:10 PM, Shawn Heisey <[hidden email]> wrote:
>
> > I've been working with Oracle employees to find better GC tuning
> > options.  The results are good enough to share with the community:
> >
> > https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning
> >
> > With the latest Java 7 or Java 8 version, and a couple of tuning
> > options, G1GC has grown up enough to be a viable choice.  Two of the
> > settings on that list were critical for making the performance
> > acceptable with my testing: ParallelRefProcEnabled and G1HeapRegionSize.
> >
> > I've included some notes on the wiki about how you can size the G1 heap
> > regions appropriately for your own index.
> >
> > Thanks,
> > Shawn
> >
> >
>
>
> --
> Bill Bell
> [hidden email]
> cell 720-256-8076
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Garbage Collection tuning - G1 is now a good option

Shawn Heisey-2
In reply to this post by Shawn Heisey-2
On 1/1/2015 12:10 PM, Shawn Heisey wrote:

> I've been working with Oracle employees to find better GC tuning
> options.  The results are good enough to share with the community:
>
> https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning
>
> With the latest Java 7 or Java 8 version, and a couple of tuning
> options, G1GC has grown up enough to be a viable choice.  Two of the
> settings on that list were critical for making the performance
> acceptable with my testing: ParallelRefProcEnabled and G1HeapRegionSize.
>
> I've included some notes on the wiki about how you can size the G1 heap
> regions appropriately for your own index.

A note was just recently added to the Lucene wiki, saying that the G1
collector should never be used with Lucene, because there are bugs that
might cause index corruption.  Solr uses Lucene, so this would apply.

https://wiki.apache.org/lucene-java/JavaBugs#Oracle_Java_.2F_Sun_Java_.2F_OpenJDK_Bugs

I have never had any problems with it, but perhaps I spoke too soon when
recommending G1.  Does anyone else have anything to add?

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Garbage Collection tuning - G1 is now a good option

Billnbell
It looks like 32 bit is affected.

> On 2013-08-14 08:27, Dawid Weiss wrote:
>>
>> Hi everyone,
>>
>> I am a committer to the Lucene/Solr project. We've recently hit what
>> we believe is a JIT/GC bug -- it manifests itself only when G1GC is
>> used, on a 32-bit VM:

On Fri, Jan 9, 2015 at 7:10 PM, Shawn Heisey <[hidden email]> wrote:

> On 1/1/2015 12:10 PM, Shawn Heisey wrote:
> > I've been working with Oracle employees to find better GC tuning
> > options.  The results are good enough to share with the community:
> >
> > https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning
> >
> > With the latest Java 7 or Java 8 version, and a couple of tuning
> > options, G1GC has grown up enough to be a viable choice.  Two of the
> > settings on that list were critical for making the performance
> > acceptable with my testing: ParallelRefProcEnabled and G1HeapRegionSize.
> >
> > I've included some notes on the wiki about how you can size the G1 heap
> > regions appropriately for your own index.
>
> A note was just recently added to the Lucene wiki, saying that the G1
> collector should never be used with Lucene, because there are bugs that
> might cause index corruption.  Solr uses Lucene, so this would apply.
>
>
> https://wiki.apache.org/lucene-java/JavaBugs#Oracle_Java_.2F_Sun_Java_.2F_OpenJDK_Bugs
>
> I have never had any problems with it, but perhaps I spoke too soon when
> recommending G1.  Does anyone else have anything to add?
>
> Thanks,
> Shawn
>
>


--
Bill Bell
[hidden email]
cell 720-256-8076
Loading...