high cpu threads (solr 7.5)

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

high cpu threads (solr 7.5)

Hari Nakka-2
We are noticing high CPU utilization on below threads.  Looks like a known
issue with. (https://github.com/netty/netty/issues/327)

But not sure if this has been addressed in any of the 1.8 releases.

Can anyone help with this?


Version: solr cloud 7.5

OS: CentOS 7

JDK: Oracle JDK 1.8.0_191





"qtp574568002-3821728" #3821728 prio=5 os_prio=0 tid=0x00007f4f20018000
nid=0x4996 runnable [0x00007f51fc6d8000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x000000064cded430> (a sun.nio.ch.Util$3)

        - locked <0x000000064cded418> (a
java.util.Collections$UnmodifiableSet)

        - locked <0x000000064cdf6e38> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)

        at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)

        at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)

        at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)

        at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)

        at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)

        at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)

        at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)

        at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)

        at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)

        at java.lang.Thread.run(Thread.java:748)
Reply | Threaded
Open this post in threaded view
|

Re: high cpu threads (solr 7.5)

Erick Erickson
It hasn’t been addressed by any Java 8 releases that I know of.

See: https://issues.apache.org/jira/browse/SOLR-13349

The work-around in Solr is trivial, see the patch so it’d be simple to patch/compile on your own.

It will be released in a Solr 7.7.2 and Solr 8.1 or later, neither of which have been released yet.

Or move to Java 9 or later.

> On Apr 3, 2019, at 4:39 PM, Hari Nakka <[hidden email]> wrote:
>
> We are noticing high CPU utilization on below threads.  Looks like a known
> issue with. (https://github.com/netty/netty/issues/327)
>
> But not sure if this has been addressed in any of the 1.8 releases.
>
> Can anyone help with this?
>
>
> Version: solr cloud 7.5
>
> OS: CentOS 7
>
> JDK: Oracle JDK 1.8.0_191
>
>
>
>
>
> "qtp574568002-3821728" #3821728 prio=5 os_prio=0 tid=0x00007f4f20018000
> nid=0x4996 runnable [0x00007f51fc6d8000]
>
>   java.lang.Thread.State: RUNNABLE
>
>        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>
>        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>
>        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>
>        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>
>        - locked <0x000000064cded430> (a sun.nio.ch.Util$3)
>
>        - locked <0x000000064cded418> (a
> java.util.Collections$UnmodifiableSet)
>
>        - locked <0x000000064cdf6e38> (a sun.nio.ch.EPollSelectorImpl)
>
>        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>
>        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>
>        at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
>
>        at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
>
>        at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
>
>        at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
>
>        at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>
>        at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>
>        at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>
>        at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
>
>        at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
>
>        at java.lang.Thread.run(Thread.java:748)

Reply | Threaded
Open this post in threaded view
|

Re: high cpu threads (solr 7.5)

Hari Nakka-2
Thank you. We are planning to upgrade the JDK 11.
Is solr 7.5 fully compatible with openjdk 11.


On Thu, Apr 4, 2019 at 9:58 AM Erick Erickson <[hidden email]>
wrote:

> It hasn’t been addressed by any Java 8 releases that I know of.
>
> See: https://issues.apache.org/jira/browse/SOLR-13349
>
> The work-around in Solr is trivial, see the patch so it’d be simple to
> patch/compile on your own.
>
> It will be released in a Solr 7.7.2 and Solr 8.1 or later, neither of
> which have been released yet.
>
> Or move to Java 9 or later.
>
> > On Apr 3, 2019, at 4:39 PM, Hari Nakka <[hidden email]> wrote:
> >
> > We are noticing high CPU utilization on below threads.  Looks like a
> known
> > issue with. (https://github.com/netty/netty/issues/327)
> >
> > But not sure if this has been addressed in any of the 1.8 releases.
> >
> > Can anyone help with this?
> >
> >
> > Version: solr cloud 7.5
> >
> > OS: CentOS 7
> >
> > JDK: Oracle JDK 1.8.0_191
> >
> >
> >
> >
> >
> > "qtp574568002-3821728" #3821728 prio=5 os_prio=0 tid=0x00007f4f20018000
> > nid=0x4996 runnable [0x00007f51fc6d8000]
> >
> >   java.lang.Thread.State: RUNNABLE
> >
> >        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >
> >        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >
> >        at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> >
> >        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >
> >        - locked <0x000000064cded430> (a sun.nio.ch.Util$3)
> >
> >        - locked <0x000000064cded418> (a
> > java.util.Collections$UnmodifiableSet)
> >
> >        - locked <0x000000064cdf6e38> (a sun.nio.ch.EPollSelectorImpl)
> >
> >        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >
> >        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> >
> >        at
> > org.eclipse.jetty.io
> .ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> >
> >        at
> > org.eclipse.jetty.io
> .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> >
> >        at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> >
> >        at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> >
> >        at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> >
> >        at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> >
> >        at
> >
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> >
> >        at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> >
> >        at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> >
> >        at java.lang.Thread.run(Thread.java:748)
>
>
Reply | Threaded
Open this post in threaded view
|

Re: high cpu threads (solr 7.5)

Erick Erickson
Here’s what the current state of various Solr x Java versions:
https://wiki.apache.org/solr/SolrJavaVersions

> On Apr 5, 2019, at 3:16 PM, Hari Nakka <[hidden email]> wrote:
>
> Thank you. We are planning to upgrade the JDK 11.
> Is solr 7.5 fully compatible with openjdk 11.
>
>
> On Thu, Apr 4, 2019 at 9:58 AM Erick Erickson <[hidden email]>
> wrote:
>
>> It hasn’t been addressed by any Java 8 releases that I know of.
>>
>> See: https://issues.apache.org/jira/browse/SOLR-13349
>>
>> The work-around in Solr is trivial, see the patch so it’d be simple to
>> patch/compile on your own.
>>
>> It will be released in a Solr 7.7.2 and Solr 8.1 or later, neither of
>> which have been released yet.
>>
>> Or move to Java 9 or later.
>>
>>> On Apr 3, 2019, at 4:39 PM, Hari Nakka <[hidden email]> wrote:
>>>
>>> We are noticing high CPU utilization on below threads.  Looks like a
>> known
>>> issue with. (https://github.com/netty/netty/issues/327)
>>>
>>> But not sure if this has been addressed in any of the 1.8 releases.
>>>
>>> Can anyone help with this?
>>>
>>>
>>> Version: solr cloud 7.5
>>>
>>> OS: CentOS 7
>>>
>>> JDK: Oracle JDK 1.8.0_191
>>>
>>>
>>>
>>>
>>>
>>> "qtp574568002-3821728" #3821728 prio=5 os_prio=0 tid=0x00007f4f20018000
>>> nid=0x4996 runnable [0x00007f51fc6d8000]
>>>
>>>  java.lang.Thread.State: RUNNABLE
>>>
>>>       at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>
>>>       at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>
>>>       at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>>
>>>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>
>>>       - locked <0x000000064cded430> (a sun.nio.ch.Util$3)
>>>
>>>       - locked <0x000000064cded418> (a
>>> java.util.Collections$UnmodifiableSet)
>>>
>>>       - locked <0x000000064cdf6e38> (a sun.nio.ch.EPollSelectorImpl)
>>>
>>>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>
>>>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>>
>>>       at
>>> org.eclipse.jetty.io
>> .ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
>>>
>>>       at
>>> org.eclipse.jetty.io
>> .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
>>>
>>>       at
>>>
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
>>>
>>>       at
>>>
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
>>>
>>>       at
>>>
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>>>
>>>       at
>>>
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>>>
>>>       at
>>>
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>>>
>>>       at
>>>
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
>>>
>>>       at
>>>
>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
>>>
>>>       at java.lang.Thread.run(Thread.java:748)
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: high cpu threads (solr 7.5)

Hari Nakka-2
Hi Erick, We upgraded JDK to 11. No improvement. Still seeing high cpu utilization randomly.  
Attached the full threaddump (tdump.out)  and lwp utilization (high-cpu.out)
there were more than 30 threads (high-cpu-dump.out)taking high cpu. these are different threads. i couldn't find much looking at them.
Can you take a look and see if there is any known condition that could be causing  this.


On Sat, Apr 6, 2019 at 12:23 PM Erick Erickson <[hidden email]> wrote:
Here’s what the current state of various Solr x Java versions:
https://wiki.apache.org/solr/SolrJavaVersions

> On Apr 5, 2019, at 3:16 PM, Hari Nakka <[hidden email]> wrote:
>
> Thank you. We are planning to upgrade the JDK 11.
> Is solr 7.5 fully compatible with openjdk 11.
>
>
> On Thu, Apr 4, 2019 at 9:58 AM Erick Erickson <[hidden email]>
> wrote:
>
>> It hasn’t been addressed by any Java 8 releases that I know of.
>>
>> See: https://issues.apache.org/jira/browse/SOLR-13349
>>
>> The work-around in Solr is trivial, see the patch so it’d be simple to
>> patch/compile on your own.
>>
>> It will be released in a Solr 7.7.2 and Solr 8.1 or later, neither of
>> which have been released yet.
>>
>> Or move to Java 9 or later.
>>
>>> On Apr 3, 2019, at 4:39 PM, Hari Nakka <[hidden email]> wrote:
>>>
>>> We are noticing high CPU utilization on below threads.  Looks like a
>> known
>>> issue with. (https://github.com/netty/netty/issues/327)
>>>
>>> But not sure if this has been addressed in any of the 1.8 releases.
>>>
>>> Can anyone help with this?
>>>
>>>
>>> Version: solr cloud 7.5
>>>
>>> OS: CentOS 7
>>>
>>> JDK: Oracle JDK 1.8.0_191
>>>
>>>
>>>
>>>
>>>
>>> "qtp574568002-3821728" #3821728 prio=5 os_prio=0 tid=0x00007f4f20018000
>>> nid=0x4996 runnable [0x00007f51fc6d8000]
>>>
>>>  java.lang.Thread.State: RUNNABLE
>>>
>>>       at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>
>>>       at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>
>>>       at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>>>
>>>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>
>>>       - locked <0x000000064cded430> (a sun.nio.ch.Util$3)
>>>
>>>       - locked <0x000000064cded418> (a
>>> java.util.Collections$UnmodifiableSet)
>>>
>>>       - locked <0x000000064cdf6e38> (a sun.nio.ch.EPollSelectorImpl)
>>>
>>>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>
>>>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>>
>>>       at
>>> org.eclipse.jetty.io
>> .ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
>>>
>>>       at
>>> org.eclipse.jetty.io
>> .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
>>>
>>>       at
>>>
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
>>>
>>>       at
>>>
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
>>>
>>>       at
>>>
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>>>
>>>       at
>>>
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>>>
>>>       at
>>>
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>>>
>>>       at
>>>
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
>>>
>>>       at
>>>
>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
>>>
>>>       at java.lang.Thread.run(Thread.java:748)
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: high cpu threads (solr 7.5)

Shawn Heisey-2
On 4/11/2019 2:21 AM, Hari Nakka wrote:
> Hi Erick, We upgraded JDK to 11. No improvement. Still seeing high cpu
> utilization randomly.
> Attached the full threaddump (tdump.out)  and lwp utilization (high-cpu.out)
> there were more than 30 threads (high-cpu-dump.out)taking high cpu.
> these are different threads. i couldn't find much looking at them.
> Can you take a look and see if there is any known condition that could
> be causing  this.

Maybe we need a prominent note on the "mailing lists" page about
attachments to the mailing list, saying something like the following:

---
Attachments rarely make it to the list.  They are stripped by the
mailing list software.  If you have files to share with the list, you'll
need to find another way to send them.  File sharing websites usually
work well.
---

We didn't get any attachments from you.

You said you included something called "lwp utilization" ... the only
lwp I have ever heard of is a perl module for HTTP, which would have
nothing at all to do with high CPU usage in Solr.

Thread dumps are usually not helpful in diagnosing high CPU usage.  They
contain zero information about which threads are consuming the most CPU.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: high cpu threads (solr 7.5)

Hari Nakka-2
I mean the light weight processes (lwp) which were taking high cpu. I
pulled the actual threads taking high cpu.
full thread dump: *tdump.out*
linux lwps: *high-cpu.out*
top high cpu lwps mapped to thread nid:  *high-cpu-dump.out
 (included threads taking more than 50% virtual core cpu)*

https://drive.google.com/drive/folders/1FsFb8l0u5ZV4qnO-tgqa91JPHqT0lCcR


On Thu, Apr 11, 2019 at 1:03 PM Shawn Heisey <[hidden email]> wrote:

> On 4/11/2019 2:21 AM, Hari Nakka wrote:
> > Hi Erick, We upgraded JDK to 11. No improvement. Still seeing high cpu
> > utilization randomly.
> > Attached the full threaddump (tdump.out)  and lwp utilization
> (high-cpu.out)
> > there were more than 30 threads (high-cpu-dump.out)taking high cpu.
> > these are different threads. i couldn't find much looking at them.
> > Can you take a look and see if there is any known condition that could
> > be causing  this.
>
> Maybe we need a prominent note on the "mailing lists" page about
> attachments to the mailing list, saying something like the following:
>
> ---
> Attachments rarely make it to the list.  They are stripped by the
> mailing list software.  If you have files to share with the list, you'll
> need to find another way to send them.  File sharing websites usually
> work well.
> ---
>
> We didn't get any attachments from you.
>
> You said you included something called "lwp utilization" ... the only
> lwp I have ever heard of is a perl module for HTTP, which would have
> nothing at all to do with high CPU usage in Solr.
>
> Thread dumps are usually not helpful in diagnosing high CPU usage.  They
> contain zero information about which threads are consuming the most CPU.
>
> Thanks,
> Shawn
>
Reply | Threaded
Open this post in threaded view
|

Re: high cpu threads (solr 7.5)

Shawn Heisey-2
On 4/11/2019 1:03 PM, Hari Nakka wrote:
> I mean the light weight processes (lwp) which were taking high cpu. I
> pulled the actual threads taking high cpu.
> full thread dump: *tdump.out*
> linux lwps: *high-cpu.out*
> top high cpu lwps mapped to thread nid:  *high-cpu-dump.out
>   (included threads taking more than 50% virtual core cpu)*

I don't know how to read any of the files you shared and come to any
useful conclusions.  I did notice that the thread names in the
"high-cpu.out" file are truncated to 15 characters, so it is not
possible to map them to the thread dump.

----------------
If I look at the "high-cpu-tdump.out" file and find threads in that dump
which are actually running (not waiting), here's what I see:

Threads that are doing lookups in one of Solr's caches, probably the
documentCache, but I am not sure:
qtp1650813924-32709

Not sure what the thread is doing, but it's running Lucene code:
qtp1650813924-54372
qtp1650813924-30839
qtp1650813924-44304
qtp1650813924-54344
qtp1650813924-40150
qtp1650813924-46829
qtp1650813924-41093

Threads that are doing a Pivot facet (Solr code):
qtp1650813924-43735
qtp1650813924-51360

Threads where Jetty is waiting for a request:
qtp1650813924-43734
qtp1650813924-52846
qtp1650813924-46575
qtp1650813924-55623
qtp1650813924-45996
qtp1650813924-48982

If the "jetty waiting" threads are causing the high CPU, then upgrading
Java should have taken care of it.  If the Lucene or Pivot Facet threads
are to blame, then you may be facing a general performance problem,
which additional memory MIGHT help with.

----------------

What requests are you sending to Solr when high CPU happens?  How long
does the high CPU last?

It might be useful to gather the screenshot described at the following
web page:

https://wiki.apache.org/solr/SolrPerformanceProblems#Asking_for_help_on_a_memory.2Fperformance_issue

One piece of information that could be useful and will not show up on
the screenshot is how many documents are being handled by that Solr
instance.  You will need to look at all of the index cores in that
instance and add up the "maxDoc" numbers for each.  The maxDoc number
includes deleted docs, which do affect performance.

Thanks,
Shawn