We have a cause for high CPU usage on Solr 7.7

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

We have a cause for high CPU usage on Solr 7.7

Erick Erickson
From the user’s list (thanks to Adam Guthrie and Lukas Weiss)

Short form: Apparently there’s a java 8 bug that’s the root cause, not fixed until Java 9: https://bugs.openjdk.java.net/browse/JDK-8129861

I have _not_ verified this, but we’ve had sporadic reports (Shawn?) of same. CommitTracker is the only place in the code that calls newScheduledThreadPool with 0.

From what I can tell, this isn’t fixed in Java until 9.

If the fix is just changing that call to use 1, it’s trivial but would require a respin of 7.7 to get out in the field.

Should I raise a JIRA? ‘cause we should be seeing it in Solr 8 and master too. Although we should be seeing this in Solr 8 too, although it should disappear in Master when we require Java 11.



************E-mail below.
Hi,
Apologies, I can’t figure out how to reply to the Solr mailing list.
I just ran across the same high CPU usage issue. I believe it’’s caused by
this commit which was introduced in Solr 7.7.0
https://github.com/apache/lucene-solr/commit/eb652b84edf441d8369f5188cdd5e3ae2b151434#diff-e54b251d166135a1afb7938cfe152bb5
There is a bug in JDK versions <=8 where using 0 threads in the
ScheduledThreadPool causes high CPU usage:
https://bugs.openjdk.java.net/browse/JDK-8129861
Oddly, the latest version
of solr/core/src/java/org/apache/solr/update/CommitTracker.java on
master still uses 0 executors as the default. Presumably most everyone is
using JDK 9 or greater which has the bug fixed, so they don’t experience
the bug.
Feel free to relay this back to the mailing list.
Thanks,
Adam Guthrie


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: We have a cause for high CPU usage on Solr 7.7

Bram Van Dam
On 25/03/2019 17:04, Erick Erickson wrote:
> Short form: Apparently there’s a java 8 bug that’s the root cause, not fixed until Java 9: https://bugs.openjdk.java.net/browse/JDK-8129861

Thanks for this, Erick & Adam. This explains a few things I've been
seeing (in my own code as well as sporadically in Solr 7.7.0). Looks
like a couple of other users on the -user list reported similar issues.

It looks like there are already a couple of bugfixes scheduled for 7.7.2
anyway, so I guess someone was already planning a new release?

Anything I can do to help out?

 - Bram

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: We have a cause for high CPU usage on Solr 7.7

Adrien Grand
In reply to this post by Erick Erickson
+1 to raise a JIRA. There is an easy workaround on the user side by
using a recent JVM, but at the same time the change to the code looks
safe. Does it affect 6.6? I see there is an ongoing thread about
getting 6.6.6 out.

On Mon, Mar 25, 2019 at 5:05 PM Erick Erickson <[hidden email]> wrote:

>
> From the user’s list (thanks to Adam Guthrie and Lukas Weiss)
>
> Short form: Apparently there’s a java 8 bug that’s the root cause, not fixed until Java 9: https://bugs.openjdk.java.net/browse/JDK-8129861
>
> I have _not_ verified this, but we’ve had sporadic reports (Shawn?) of same. CommitTracker is the only place in the code that calls newScheduledThreadPool with 0.
>
> From what I can tell, this isn’t fixed in Java until 9.
>
> If the fix is just changing that call to use 1, it’s trivial but would require a respin of 7.7 to get out in the field.
>
> Should I raise a JIRA? ‘cause we should be seeing it in Solr 8 and master too. Although we should be seeing this in Solr 8 too, although it should disappear in Master when we require Java 11.
>
>
>
> ************E-mail below.
> Hi,
> Apologies, I can’t figure out how to reply to the Solr mailing list.
> I just ran across the same high CPU usage issue. I believe it’’s caused by
> this commit which was introduced in Solr 7.7.0
> https://github.com/apache/lucene-solr/commit/eb652b84edf441d8369f5188cdd5e3ae2b151434#diff-e54b251d166135a1afb7938cfe152bb5
> There is a bug in JDK versions <=8 where using 0 threads in the
> ScheduledThreadPool causes high CPU usage:
> https://bugs.openjdk.java.net/browse/JDK-8129861
> Oddly, the latest version
> of solr/core/src/java/org/apache/solr/update/CommitTracker.java on
> master still uses 0 executors as the default. Presumably most everyone is
> using JDK 9 or greater which has the bug fixed, so they don’t experience
> the bug.
> Feel free to relay this back to the mailing list.
> Thanks,
> Adam Guthrie
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


--
Adrien

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: We have a cause for high CPU usage on Solr 7.7

Ishan Chattopadhyaya
This is now tracked in SOLR-13349. I don't think 6.6 is affected by this.

On Tue, Mar 26, 2019 at 3:50 PM Adrien Grand <[hidden email]> wrote:
+1 to raise a JIRA. There is an easy workaround on the user side by
using a recent JVM, but at the same time the change to the code looks
safe. Does it affect 6.6? I see there is an ongoing thread about
getting 6.6.6 out.

On Mon, Mar 25, 2019 at 5:05 PM Erick Erickson <[hidden email]> wrote:
>
> From the user’s list (thanks to Adam Guthrie and Lukas Weiss)
>
> Short form: Apparently there’s a java 8 bug that’s the root cause, not fixed until Java 9: https://bugs.openjdk.java.net/browse/JDK-8129861
>
> I have _not_ verified this, but we’ve had sporadic reports (Shawn?) of same. CommitTracker is the only place in the code that calls newScheduledThreadPool with 0.
>
> From what I can tell, this isn’t fixed in Java until 9.
>
> If the fix is just changing that call to use 1, it’s trivial but would require a respin of 7.7 to get out in the field.
>
> Should I raise a JIRA? ‘cause we should be seeing it in Solr 8 and master too. Although we should be seeing this in Solr 8 too, although it should disappear in Master when we require Java 11.
>
>
>
> ************E-mail below.
> Hi,
> Apologies, I can’t figure out how to reply to the Solr mailing list.
> I just ran across the same high CPU usage issue. I believe it’’s caused by
> this commit which was introduced in Solr 7.7.0
> https://github.com/apache/lucene-solr/commit/eb652b84edf441d8369f5188cdd5e3ae2b151434#diff-e54b251d166135a1afb7938cfe152bb5
> There is a bug in JDK versions <=8 where using 0 threads in the
> ScheduledThreadPool causes high CPU usage:
> https://bugs.openjdk.java.net/browse/JDK-8129861
> Oddly, the latest version
> of solr/core/src/java/org/apache/solr/update/CommitTracker.java on
> master still uses 0 executors as the default. Presumably most everyone is
> using JDK 9 or greater which has the bug fixed, so they don’t experience
> the bug.
> Feel free to relay this back to the mailing list.
> Thanks,
> Adam Guthrie
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


--
Adrien

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]