significant lucene benchmark regression: JDK11?

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

significant lucene benchmark regression: JDK11?

Nicholas Knize
Earlier this week I noticed a significant across the board performance regression on the nightly geo benchmarks. It appears this regression can also be seen on other lucene benchmarks and appears to correspond to the upgrade to JDK 11.

Any thoughts?

Nicholas Knize, Ph.D., GISP
Geospatial Software Guy  |  Elasticsearch
Apache Lucene PMC Member and Committer
[hidden email]  
Reply | Threaded
Open this post in threaded view
|

Re: significant lucene benchmark regression: JDK11?

Robert Muir
looks to me like the default garbage collector may play a part in
this? look at JIT/gc times

https://home.apache.org/~mikemccand/lucenebench/indexing.html

On Thu, Apr 25, 2019 at 1:57 PM Nicholas Knize <[hidden email]> wrote:
>
> Earlier this week I noticed a significant across the board performance regression on the nightly geo benchmarks. It appears this regression can also be seen on other lucene benchmarks and appears to correspond to the upgrade to JDK 11.
>
> Any thoughts?
>
> Nicholas Knize, Ph.D., GISP
> Geospatial Software Guy  |  Elasticsearch
> Apache Lucene PMC Member and Committer
> [hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: significant lucene benchmark regression: JDK11?

Michael Sokolov-4
Strangely LatLonShape seems to move in the opposite direction, or was
that due to a known functional change?

On Thu, Apr 25, 2019 at 3:33 PM Robert Muir <[hidden email]> wrote:

>
> looks to me like the default garbage collector may play a part in
> this? look at JIT/gc times
>
> https://home.apache.org/~mikemccand/lucenebench/indexing.html
>
> On Thu, Apr 25, 2019 at 1:57 PM Nicholas Knize <[hidden email]> wrote:
> >
> > Earlier this week I noticed a significant across the board performance regression on the nightly geo benchmarks. It appears this regression can also be seen on other lucene benchmarks and appears to correspond to the upgrade to JDK 11.
> >
> > Any thoughts?
> >
> > Nicholas Knize, Ph.D., GISP
> > Geospatial Software Guy  |  Elasticsearch
> > Apache Lucene PMC Member and Committer
> > [hidden email]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: significant lucene benchmark regression: JDK11?

Uwe Schindler
In reply to this post by Nicholas Knize
Hi,

I am not sure how Mike's benchmarks are setup and if he chooses a specific garbage collector.

Java 8 defaults to ParallelGC, Java 11 defaults to G1, which may slow down up to 10% as it is not optimized for throughput.

So to compare you gave to be specific in your GC choices.

Uwe

Am April 25, 2019 5:57:16 PM UTC schrieb Nicholas Knize <[hidden email]>:
Earlier this week I noticed a significant across the board performance regression on the nightly geo benchmarks. It appears this regression can also be seen on other lucene benchmarks and appears to correspond to the upgrade to JDK 11.

Any thoughts?

Nicholas Knize, Ph.D., GISP
Geospatial Software Guy  |  Elasticsearch
Apache Lucene PMC Member and Committer
[hidden email]  

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
Reply | Threaded
Open this post in threaded view
|

Re: significant lucene benchmark regression: JDK11?

Michael McCandless-2
Yeah I'm just using the JDK's default in the nightly benchmarks.

Should I override back to the parallel collector?

On Thu, Apr 25, 2019 at 5:44 PM Uwe Schindler <[hidden email]> wrote:
Hi,

I am not sure how Mike's benchmarks are setup and if he chooses a specific garbage collector.

Java 8 defaults to ParallelGC, Java 11 defaults to G1, which may slow down up to 10% as it is not optimized for throughput.

So to compare you gave to be specific in your GC choices.

Uwe

Am April 25, 2019 5:57:16 PM UTC schrieb Nicholas Knize <[hidden email]>:
Earlier this week I noticed a significant across the board performance regression on the nightly geo benchmarks. It appears this regression can also be seen on other lucene benchmarks and appears to correspond to the upgrade to JDK 11.

Any thoughts?

Nicholas Knize, Ph.D., GISP
Geospatial Software Guy  |  Elasticsearch
Apache Lucene PMC Member and Committer
[hidden email]  

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
Reply | Threaded
Open this post in threaded view
|

Re: significant lucene benchmark regression: JDK11?

Uwe Schindler
Maybe do this temporary, to not have 2 changes at the same time.

Uwe

Am April 25, 2019 9:48:35 PM UTC schrieb Michael McCandless <[hidden email]>:
Yeah I'm just using the JDK's default in the nightly benchmarks.

Should I override back to the parallel collector?

On Thu, Apr 25, 2019 at 5:44 PM Uwe Schindler <[hidden email]> wrote:
Hi,

I am not sure how Mike's benchmarks are setup and if he chooses a specific garbage collector.

Java 8 defaults to ParallelGC, Java 11 defaults to G1, which may slow down up to 10% as it is not optimized for throughput.

So to compare you gave to be specific in your GC choices.

Uwe

Am April 25, 2019 5:57:16 PM UTC schrieb Nicholas Knize <[hidden email]>:
Earlier this week I noticed a significant across the board performance regression on the nightly geo benchmarks. It appears this regression can also be seen on other lucene benchmarks and appears to correspond to the upgrade to JDK 11.

Any thoughts?

Nicholas Knize, Ph.D., GISP
Geospatial Software Guy  |  Elasticsearch
Apache Lucene PMC Member and Committer
[hidden email]  

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
Reply | Threaded
Open this post in threaded view
|

Re: significant lucene benchmark regression: JDK11?

david.w.smiley@gmail.com
+1 to choose the Parallel Collector temporarily as Uwe suggests so we can see distinct effects of the separate changes.

I suppose the benchmarks should continue to prefer the default JVM settings, as it is how users will consume Lucene.

~ David Smiley
Apache Lucene/Solr Search Developer


On Thu, Apr 25, 2019 at 6:03 PM Uwe Schindler <[hidden email]> wrote:
Maybe do this temporary, to not have 2 changes at the same time.

Uwe

Am April 25, 2019 9:48:35 PM UTC schrieb Michael McCandless <[hidden email]>:
Yeah I'm just using the JDK's default in the nightly benchmarks.

Should I override back to the parallel collector?

On Thu, Apr 25, 2019 at 5:44 PM Uwe Schindler <[hidden email]> wrote:
Hi,

I am not sure how Mike's benchmarks are setup and if he chooses a specific garbage collector.

Java 8 defaults to ParallelGC, Java 11 defaults to G1, which may slow down up to 10% as it is not optimized for throughput.

So to compare you gave to be specific in your GC choices.

Uwe

Am April 25, 2019 5:57:16 PM UTC schrieb Nicholas Knize <[hidden email]>:
Earlier this week I noticed a significant across the board performance regression on the nightly geo benchmarks. It appears this regression can also be seen on other lucene benchmarks and appears to correspond to the upgrade to JDK 11.

Any thoughts?

Nicholas Knize, Ph.D., GISP
Geospatial Software Guy  |  Elasticsearch
Apache Lucene PMC Member and Committer
[hidden email]  

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
Reply | Threaded
Open this post in threaded view
|

Re: significant lucene benchmark regression: JDK11?

Michael McCandless-2
OK I'll make this change soon and reply back.

On Fri, Apr 26, 2019 at 10:17 AM David Smiley <[hidden email]> wrote:
+1 to choose the Parallel Collector temporarily as Uwe suggests so we can see distinct effects of the separate changes.

I suppose the benchmarks should continue to prefer the default JVM settings, as it is how users will consume Lucene.

~ David Smiley
Apache Lucene/Solr Search Developer


On Thu, Apr 25, 2019 at 6:03 PM Uwe Schindler <[hidden email]> wrote:
Maybe do this temporary, to not have 2 changes at the same time.

Uwe

Am April 25, 2019 9:48:35 PM UTC schrieb Michael McCandless <[hidden email]>:
Yeah I'm just using the JDK's default in the nightly benchmarks.

Should I override back to the parallel collector?

On Thu, Apr 25, 2019 at 5:44 PM Uwe Schindler <[hidden email]> wrote:
Hi,

I am not sure how Mike's benchmarks are setup and if he chooses a specific garbage collector.

Java 8 defaults to ParallelGC, Java 11 defaults to G1, which may slow down up to 10% as it is not optimized for throughput.

So to compare you gave to be specific in your GC choices.

Uwe

Am April 25, 2019 5:57:16 PM UTC schrieb Nicholas Knize <[hidden email]>:
Earlier this week I noticed a significant across the board performance regression on the nightly geo benchmarks. It appears this regression can also be seen on other lucene benchmarks and appears to correspond to the upgrade to JDK 11.

Any thoughts?

Nicholas Knize, Ph.D., GISP
Geospatial Software Guy  |  Elasticsearch
Apache Lucene PMC Member and Committer
[hidden email]  

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
Reply | Threaded
Open this post in threaded view
|

Re: significant lucene benchmark regression: JDK11?

Michael McCandless-2
Hi Team,

OK I fixed nightly bench to specify ParallelGC for everything, and indeed that recovered many of the losses -- G1 sure is costly.  I added annotations to the charts too.

But some things still have losses, e.g. Russia Polygon filter is looking bad: https://home.apache.org/~mikemccand/geobench.html#search-polyRussia

On Mon, Apr 29, 2019 at 1:48 PM Michael McCandless <[hidden email]> wrote:
OK I'll make this change soon and reply back.

On Fri, Apr 26, 2019 at 10:17 AM David Smiley <[hidden email]> wrote:
+1 to choose the Parallel Collector temporarily as Uwe suggests so we can see distinct effects of the separate changes.

I suppose the benchmarks should continue to prefer the default JVM settings, as it is how users will consume Lucene.

~ David Smiley
Apache Lucene/Solr Search Developer


On Thu, Apr 25, 2019 at 6:03 PM Uwe Schindler <[hidden email]> wrote:
Maybe do this temporary, to not have 2 changes at the same time.

Uwe

Am April 25, 2019 9:48:35 PM UTC schrieb Michael McCandless <[hidden email]>:
Yeah I'm just using the JDK's default in the nightly benchmarks.

Should I override back to the parallel collector?

On Thu, Apr 25, 2019 at 5:44 PM Uwe Schindler <[hidden email]> wrote:
Hi,

I am not sure how Mike's benchmarks are setup and if he chooses a specific garbage collector.

Java 8 defaults to ParallelGC, Java 11 defaults to G1, which may slow down up to 10% as it is not optimized for throughput.

So to compare you gave to be specific in your GC choices.

Uwe

Am April 25, 2019 5:57:16 PM UTC schrieb Nicholas Knize <[hidden email]>:
Earlier this week I noticed a significant across the board performance regression on the nightly geo benchmarks. It appears this regression can also be seen on other lucene benchmarks and appears to correspond to the upgrade to JDK 11.

Any thoughts?

Nicholas Knize, Ph.D., GISP
Geospatial Software Guy  |  Elasticsearch
Apache Lucene PMC Member and Committer
[hidden email]  

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
Reply | Threaded
Open this post in threaded view
|

Re: significant lucene benchmark regression: JDK11?

Adrien Grand
The Russia polygon seems to be something else? It slowed down on April
12th and then a bit more on April 20th while you only switched to
OpenJDK11 on April 23th?

However some other slowdowns seem to happen at the same time as the
switch to OpenJDK 11 and haven't been addressed by the change of
garbage collector such as sorting tasks ( see e.g.
https://home.apache.org/~mikemccand/lucenebench/TermDTSort.html).

On Tue, Apr 30, 2019 at 3:30 PM Michael McCandless
<[hidden email]> wrote:

>
> Hi Team,
>
> OK I fixed nightly bench to specify ParallelGC for everything, and indeed that recovered many of the losses -- G1 sure is costly.  I added annotations to the charts too.
>
> But some things still have losses, e.g. Russia Polygon filter is looking bad: https://home.apache.org/~mikemccand/geobench.html#search-polyRussia
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Mon, Apr 29, 2019 at 1:48 PM Michael McCandless <[hidden email]> wrote:
>>
>> OK I'll make this change soon and reply back.
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Fri, Apr 26, 2019 at 10:17 AM David Smiley <[hidden email]> wrote:
>>>
>>> +1 to choose the Parallel Collector temporarily as Uwe suggests so we can see distinct effects of the separate changes.
>>>
>>> I suppose the benchmarks should continue to prefer the default JVM settings, as it is how users will consume Lucene.
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>> On Thu, Apr 25, 2019 at 6:03 PM Uwe Schindler <[hidden email]> wrote:
>>>>
>>>> Maybe do this temporary, to not have 2 changes at the same time.
>>>>
>>>> Uwe
>>>>
>>>> Am April 25, 2019 9:48:35 PM UTC schrieb Michael McCandless <[hidden email]>:
>>>>>
>>>>> Yeah I'm just using the JDK's default in the nightly benchmarks.
>>>>>
>>>>> Should I override back to the parallel collector?
>>>>>
>>>>> Mike McCandless
>>>>>
>>>>> http://blog.mikemccandless.com
>>>>>
>>>>>
>>>>> On Thu, Apr 25, 2019 at 5:44 PM Uwe Schindler <[hidden email]> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am not sure how Mike's benchmarks are setup and if he chooses a specific garbage collector.
>>>>>>
>>>>>> Java 8 defaults to ParallelGC, Java 11 defaults to G1, which may slow down up to 10% as it is not optimized for throughput.
>>>>>>
>>>>>> So to compare you gave to be specific in your GC choices.
>>>>>>
>>>>>> Uwe
>>>>>>
>>>>>> Am April 25, 2019 5:57:16 PM UTC schrieb Nicholas Knize <[hidden email]>:
>>>>>>>
>>>>>>> Earlier this week I noticed a significant across the board performance regression on the nightly geo benchmarks. It appears this regression can also be seen on other lucene benchmarks and appears to correspond to the upgrade to JDK 11.
>>>>>>>
>>>>>>> Any thoughts?
>>>>>>>
>>>>>>> Nicholas Knize, Ph.D., GISP
>>>>>>> Geospatial Software Guy  |  Elasticsearch
>>>>>>> Apache Lucene PMC Member and Committer
>>>>>>> [hidden email]
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Uwe Schindler
>>>>>> Achterdiek 19, 28357 Bremen
>>>>>> https://www.thetaphi.de
>>>>
>>>>
>>>> --
>>>> Uwe Schindler
>>>> Achterdiek 19, 28357 Bremen
>>>> https://www.thetaphi.de



--
Adrien

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: significant lucene benchmark regression: JDK11?

Michael McCandless-2
Yeah there are definitely more slowdowns still happening besides the GC issue!

On Fri, May 10, 2019 at 2:15 AM Adrien Grand <[hidden email]> wrote:
The Russia polygon seems to be something else? It slowed down on April
12th and then a bit more on April 20th while you only switched to
OpenJDK11 on April 23th?

However some other slowdowns seem to happen at the same time as the
switch to OpenJDK 11 and haven't been addressed by the change of
garbage collector such as sorting tasks ( see e.g.
https://home.apache.org/~mikemccand/lucenebench/TermDTSort.html).

On Tue, Apr 30, 2019 at 3:30 PM Michael McCandless
<[hidden email]> wrote:
>
> Hi Team,
>
> OK I fixed nightly bench to specify ParallelGC for everything, and indeed that recovered many of the losses -- G1 sure is costly.  I added annotations to the charts too.
>
> But some things still have losses, e.g. Russia Polygon filter is looking bad: https://home.apache.org/~mikemccand/geobench.html#search-polyRussia
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Mon, Apr 29, 2019 at 1:48 PM Michael McCandless <[hidden email]> wrote:
>>
>> OK I'll make this change soon and reply back.
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Fri, Apr 26, 2019 at 10:17 AM David Smiley <[hidden email]> wrote:
>>>
>>> +1 to choose the Parallel Collector temporarily as Uwe suggests so we can see distinct effects of the separate changes.
>>>
>>> I suppose the benchmarks should continue to prefer the default JVM settings, as it is how users will consume Lucene.
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>> On Thu, Apr 25, 2019 at 6:03 PM Uwe Schindler <[hidden email]> wrote:
>>>>
>>>> Maybe do this temporary, to not have 2 changes at the same time.
>>>>
>>>> Uwe
>>>>
>>>> Am April 25, 2019 9:48:35 PM UTC schrieb Michael McCandless <[hidden email]>:
>>>>>
>>>>> Yeah I'm just using the JDK's default in the nightly benchmarks.
>>>>>
>>>>> Should I override back to the parallel collector?
>>>>>
>>>>> Mike McCandless
>>>>>
>>>>> http://blog.mikemccandless.com
>>>>>
>>>>>
>>>>> On Thu, Apr 25, 2019 at 5:44 PM Uwe Schindler <[hidden email]> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am not sure how Mike's benchmarks are setup and if he chooses a specific garbage collector.
>>>>>>
>>>>>> Java 8 defaults to ParallelGC, Java 11 defaults to G1, which may slow down up to 10% as it is not optimized for throughput.
>>>>>>
>>>>>> So to compare you gave to be specific in your GC choices.
>>>>>>
>>>>>> Uwe
>>>>>>
>>>>>> Am April 25, 2019 5:57:16 PM UTC schrieb Nicholas Knize <[hidden email]>:
>>>>>>>
>>>>>>> Earlier this week I noticed a significant across the board performance regression on the nightly geo benchmarks. It appears this regression can also be seen on other lucene benchmarks and appears to correspond to the upgrade to JDK 11.
>>>>>>>
>>>>>>> Any thoughts?
>>>>>>>
>>>>>>> Nicholas Knize, Ph.D., GISP
>>>>>>> Geospatial Software Guy  |  Elasticsearch
>>>>>>> Apache Lucene PMC Member and Committer
>>>>>>> [hidden email]
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Uwe Schindler
>>>>>> Achterdiek 19, 28357 Bremen
>>>>>> https://www.thetaphi.de
>>>>
>>>>
>>>> --
>>>> Uwe Schindler
>>>> Achterdiek 19, 28357 Bremen
>>>> https://www.thetaphi.de



--
Adrien

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]