High CPU usage with Solr 7.7.0

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

High CPU usage with Solr 7.7.0

Lukas Weiss
Hello,

we recently updated our Solr server from 6.6.5 to 7.7.0. Since then, we
have problems with the server's CPU usage.
We have two Solr cores configured, but even if we clear all indexes and do
not start the index process, we see 100 CPU usage for both cores.

Here's what our top says:

root@solr:~ # top
top - 09:25:24 up 17:40,  1 user,  load average: 2,28, 2,56, 2,68
Threads:  74 total,   3 running,  71 sleeping,   0 stopped,   0 zombie
%Cpu0  :100,0 us,  0,0 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,0 si,
0,0 st
%Cpu1  :100,0 us,  0,0 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,0 si,
0,0 st
%Cpu2  : 11,3 us,  1,0 sy,  0,0 ni, 86,7 id,  0,7 wa,  0,0 hi,  0,3 si,
0,0 st
%Cpu3  :  3,0 us,  3,0 sy,  0,0 ni, 93,7 id,  0,3 wa,  0,0 hi,  0,0 si,
0,0 st
KiB Mem :  8388608 total,  7859168 free,   496744 used,    32696
buff/cache
KiB Swap:  2097152 total,  2097152 free,        0 used.  7859168 avail Mem


  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
              P
10209 solr      20   0 6138468 452520  25740 R 99,9  5,4  29:43.45 java
-server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
-XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 24
10214 solr      20   0 6138468 452520  25740 R 99,9  5,4  28:42.91 java
-server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
-XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 25

The solr server is installed on a Debian Stretch 9.8 (64bit) on Linux LXC
dedicated Container.

Some more server info:

root@solr:~ # java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

root@solr:~ # free -m
              total        used        free      shared  buff/cache
available
Mem:           8192         484        7675         701          31 7675
Swap:          2048           0        2048

We also found something strange if we do an strace of the main process, we
get lots of ongoing connection timeouts:

root@solr:~ # strace -F -p 4136
strace: Process 4136 attached with 48 threads
strace: [ Process PID=11089 runs in x32 mode. ]
[pid  4937] epoll_wait(139,  <unfinished ...>
[pid  4936] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4909] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4618] epoll_wait(136,  <unfinished ...>
[pid  4576] futex(0x7ff61ce66474, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
...>
[pid  4279] futex(0x7ff61ce62b34, FUTEX_WAIT_PRIVATE, 2203, NULL
<unfinished ...>
[pid  4244] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4227] futex(0x7ff56c71ae14, FUTEX_WAIT_PRIVATE, 2237, NULL
<unfinished ...>
[pid  4243] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4228] futex(0x7ff5608331a4, FUTEX_WAIT_PRIVATE, 2237, NULL
<unfinished ...>
[pid  4208] futex(0x7ff61ce63e54, FUTEX_WAIT_PRIVATE, 5, NULL <unfinished
...>
[pid  4205] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4204] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4196] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4195] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4194] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4193] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4187] restart_syscall(<... resuming interrupted restart_syscall ...>
<unfinished ...>
[pid  4180] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4179] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4177] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4174] accept(133,  <unfinished ...>
[pid  4173] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4172] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4171] restart_syscall(<... resuming interrupted restart_syscall ...>
<unfinished ...>
[pid  4165] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4164] futex(0x7ff61c1f5054, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished
...>
[pid  4163] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4162] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4161] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4160] futex(0x7ff623d52c20,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff
<unfinished ...>
[pid  4159] futex(0x7ff61c1e9d54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid  4158] futex(0x7ff61c1b7f54, FUTEX_WAIT_PRIVATE, 15, NULL <unfinished
...>
[pid  4157] futex(0x7ff61c1b5554, FUTEX_WAIT_PRIVATE, 19, NULL <unfinished
...>
[pid  4156] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4155] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4153] futex(0x7ff61c06c754, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid  4152] futex(0x7ff61c06ab54, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished
...>
[pid  4151] futex(0x7ff61c068f54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid  4150] futex(0x7ff61c067354, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid  4148] futex(0x7ff61c024a54, FUTEX_WAIT_PRIVATE, 403, NULL
<unfinished ...>
[pid  4165] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564856, tv_nsec=849859736}, 0xffffffff <unfinished ...>
[pid  4147] futex(0x7ff61c022e54, FUTEX_WAIT_PRIVATE, 415, NULL
<unfinished ...>
[pid  4146] futex(0x7ff61c021254, FUTEX_WAIT_PRIVATE, 397, NULL
<unfinished ...>
[pid  4145] futex(0x7ff61c01f654, FUTEX_WAIT_PRIVATE, 405, NULL
<unfinished ...>
[pid  4144] futex(0x7ff61c00e354, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
...>
[pid  4136] futex(0x7ff624b729d0, FUTEX_WAIT, 4144, NULL <unfinished ...>
[pid  4165] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed
out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564856, tv_nsec=900162344}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564856, tv_nsec=950365105}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=586325}, 0xffffffff) = -1 ETIMEDOUT (Connection
timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=50791977}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=100997890}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=151206817}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=201402531}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=251616284}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=301813556}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=352036802}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=402239182}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=452439835}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=502635489}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=552844020}, 0xffffffff <unfinished ...>
[pid  4156] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
timed out)
[pid  4156] futex(0x7ff61c1aba28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4156] futex(0x7ff61c1aba54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564858, tv_nsec=506449064}, 0xffffffff <unfinished ...>
[pid  4165] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed
out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=603013734}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=653149664}, 0xffffffff^Cstrace: Process 4136
detached
strace: Process 4144 detached
strace: Process 4145 detached
strace: Process 4146 detached
strace: Process 4147 detached
strace: Process 4148 detached
strace: Process 4150 detached
strace: Process 4151 detached
strace: Process 4152 detached
strace: Process 4153 detached
....


Could you help us to determine what's wrong with our setup?

Thank you very much,

Kind regards
Lukas Weiss
Reply | Threaded
Open this post in threaded view
|

Re: High CPU usage with Solr 7.7.0

Joe Obernberger
Just to add to this.  We upgraded to 7.7.0 and saw very large CPU usage
on multi core boxes - sustained in the 1200% range.  We then switched to
7.6.0 (no other configuration changes) and the problem went away.

We have a 40 node cluster and all 40 nodes had high CPU usage with 3
indexes stored on HDFS.

-Joe

On 2/27/2019 5:04 AM, Lukas Weiss wrote:

> Hello,
>
> we recently updated our Solr server from 6.6.5 to 7.7.0. Since then, we
> have problems with the server's CPU usage.
> We have two Solr cores configured, but even if we clear all indexes and do
> not start the index process, we see 100 CPU usage for both cores.
>
> Here's what our top says:
>
> root@solr:~ # top
> top - 09:25:24 up 17:40,  1 user,  load average: 2,28, 2,56, 2,68
> Threads:  74 total,   3 running,  71 sleeping,   0 stopped,   0 zombie
> %Cpu0  :100,0 us,  0,0 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,0 si,
> 0,0 st
> %Cpu1  :100,0 us,  0,0 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,0 si,
> 0,0 st
> %Cpu2  : 11,3 us,  1,0 sy,  0,0 ni, 86,7 id,  0,7 wa,  0,0 hi,  0,3 si,
> 0,0 st
> %Cpu3  :  3,0 us,  3,0 sy,  0,0 ni, 93,7 id,  0,3 wa,  0,0 hi,  0,0 si,
> 0,0 st
> KiB Mem :  8388608 total,  7859168 free,   496744 used,    32696
> buff/cache
> KiB Swap:  2097152 total,  2097152 free,        0 used.  7859168 avail Mem
>
>
>    PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
>                P
> 10209 solr      20   0 6138468 452520  25740 R 99,9  5,4  29:43.45 java
> -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 24
> 10214 solr      20   0 6138468 452520  25740 R 99,9  5,4  28:42.91 java
> -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 25
>
> The solr server is installed on a Debian Stretch 9.8 (64bit) on Linux LXC
> dedicated Container.
>
> Some more server info:
>
> root@solr:~ # java -version
> openjdk version "1.8.0_181"
> OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
> OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
>
> root@solr:~ # free -m
>                total        used        free      shared  buff/cache
> available
> Mem:           8192         484        7675         701          31 7675
> Swap:          2048           0        2048
>
> We also found something strange if we do an strace of the main process, we
> get lots of ongoing connection timeouts:
>
> root@solr:~ # strace -F -p 4136
> strace: Process 4136 attached with 48 threads
> strace: [ Process PID=11089 runs in x32 mode. ]
> [pid  4937] epoll_wait(139,  <unfinished ...>
> [pid  4936] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4909] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4618] epoll_wait(136,  <unfinished ...>
> [pid  4576] futex(0x7ff61ce66474, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
> ...>
> [pid  4279] futex(0x7ff61ce62b34, FUTEX_WAIT_PRIVATE, 2203, NULL
> <unfinished ...>
> [pid  4244] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4227] futex(0x7ff56c71ae14, FUTEX_WAIT_PRIVATE, 2237, NULL
> <unfinished ...>
> [pid  4243] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4228] futex(0x7ff5608331a4, FUTEX_WAIT_PRIVATE, 2237, NULL
> <unfinished ...>
> [pid  4208] futex(0x7ff61ce63e54, FUTEX_WAIT_PRIVATE, 5, NULL <unfinished
> ...>
> [pid  4205] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4204] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4196] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4195] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4194] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4193] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4187] restart_syscall(<... resuming interrupted restart_syscall ...>
> <unfinished ...>
> [pid  4180] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4179] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4177] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4174] accept(133,  <unfinished ...>
> [pid  4173] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4172] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4171] restart_syscall(<... resuming interrupted restart_syscall ...>
> <unfinished ...>
> [pid  4165] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4164] futex(0x7ff61c1f5054, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished
> ...>
> [pid  4163] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4162] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4161] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4160] futex(0x7ff623d52c20,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff
> <unfinished ...>
> [pid  4159] futex(0x7ff61c1e9d54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
> ...>
> [pid  4158] futex(0x7ff61c1b7f54, FUTEX_WAIT_PRIVATE, 15, NULL <unfinished
> ...>
> [pid  4157] futex(0x7ff61c1b5554, FUTEX_WAIT_PRIVATE, 19, NULL <unfinished
> ...>
> [pid  4156] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4155] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4153] futex(0x7ff61c06c754, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
> ...>
> [pid  4152] futex(0x7ff61c06ab54, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished
> ...>
> [pid  4151] futex(0x7ff61c068f54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
> ...>
> [pid  4150] futex(0x7ff61c067354, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
> ...>
> [pid  4148] futex(0x7ff61c024a54, FUTEX_WAIT_PRIVATE, 403, NULL
> <unfinished ...>
> [pid  4165] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=849859736}, 0xffffffff <unfinished ...>
> [pid  4147] futex(0x7ff61c022e54, FUTEX_WAIT_PRIVATE, 415, NULL
> <unfinished ...>
> [pid  4146] futex(0x7ff61c021254, FUTEX_WAIT_PRIVATE, 397, NULL
> <unfinished ...>
> [pid  4145] futex(0x7ff61c01f654, FUTEX_WAIT_PRIVATE, 405, NULL
> <unfinished ...>
> [pid  4144] futex(0x7ff61c00e354, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
> ...>
> [pid  4136] futex(0x7ff624b729d0, FUTEX_WAIT, 4144, NULL <unfinished ...>
> [pid  4165] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed
> out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=900162344}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=950365105}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=586325}, 0xffffffff) = -1 ETIMEDOUT (Connection
> timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=50791977}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=100997890}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=151206817}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=201402531}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=251616284}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=301813556}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=352036802}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=402239182}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=452439835}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=502635489}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=552844020}, 0xffffffff <unfinished ...>
> [pid  4156] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> timed out)
> [pid  4156] futex(0x7ff61c1aba28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4156] futex(0x7ff61c1aba54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564858, tv_nsec=506449064}, 0xffffffff <unfinished ...>
> [pid  4165] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed
> out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=603013734}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=653149664}, 0xffffffff^Cstrace: Process 4136
> detached
> strace: Process 4144 detached
> strace: Process 4145 detached
> strace: Process 4146 detached
> strace: Process 4147 detached
> strace: Process 4148 detached
> strace: Process 4150 detached
> strace: Process 4151 detached
> strace: Process 4152 detached
> strace: Process 4153 detached
> ....
>
>
> Could you help us to determine what's wrong with our setup?
>
> Thank you very much,
>
> Kind regards
> Lukas Weiss
>
> ---
> This email has been checked for viruses by AVG.
> https://www.avg.com
>
Reply | Threaded
Open this post in threaded view
|

Antwort: Re: High CPU usage with Solr 7.7.0

Lukas Weiss
I can confirm this. Downgrading to 7.6.0 solved the issue.
Thanks for the hint.



Von:    "Joe Obernberger" <[hidden email]>
An:     [hidden email], "Lukas Weiss"
<[hidden email]>,
Datum:  27.02.2019 15:59
Betreff:        Re: High CPU usage with Solr 7.7.0



Just to add to this.  We upgraded to 7.7.0 and saw very large CPU usage
on multi core boxes - sustained in the 1200% range.  We then switched to
7.6.0 (no other configuration changes) and the problem went away.

We have a 40 node cluster and all 40 nodes had high CPU usage with 3
indexes stored on HDFS.

-Joe

On 2/27/2019 5:04 AM, Lukas Weiss wrote:
> Hello,
>
> we recently updated our Solr server from 6.6.5 to 7.7.0. Since then, we
> have problems with the server's CPU usage.
> We have two Solr cores configured, but even if we clear all indexes and
do

> not start the index process, we see 100 CPU usage for both cores.
>
> Here's what our top says:
>
> root@solr:~ # top
> top - 09:25:24 up 17:40,  1 user,  load average: 2,28, 2,56, 2,68
> Threads:  74 total,   3 running,  71 sleeping,   0 stopped,   0 zombie
> %Cpu0  :100,0 us,  0,0 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,0 si,
> 0,0 st
> %Cpu1  :100,0 us,  0,0 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,0 si,
> 0,0 st
> %Cpu2  : 11,3 us,  1,0 sy,  0,0 ni, 86,7 id,  0,7 wa,  0,0 hi,  0,3 si,
> 0,0 st
> %Cpu3  :  3,0 us,  3,0 sy,  0,0 ni, 93,7 id,  0,3 wa,  0,0 hi,  0,0 si,
> 0,0 st
> KiB Mem :  8388608 total,  7859168 free,   496744 used,    32696
> buff/cache
> KiB Swap:  2097152 total,  2097152 free,        0 used.  7859168 avail
Mem
>
>
>    PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+
COMMAND

>                P
> 10209 solr      20   0 6138468 452520  25740 R 99,9  5,4  29:43.45 java
> -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 24
> 10214 solr      20   0 6138468 452520  25740 R 99,9  5,4  28:42.91 java
> -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 25
>
> The solr server is installed on a Debian Stretch 9.8 (64bit) on Linux
LXC

> dedicated Container.
>
> Some more server info:
>
> root@solr:~ # java -version
> openjdk version "1.8.0_181"
> OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
> OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
>
> root@solr:~ # free -m
>                total        used        free      shared  buff/cache
> available
> Mem:           8192         484        7675         701          31 7675
> Swap:          2048           0        2048
>
> We also found something strange if we do an strace of the main process,
we

> get lots of ongoing connection timeouts:
>
> root@solr:~ # strace -F -p 4136
> strace: Process 4136 attached with 48 threads
> strace: [ Process PID=11089 runs in x32 mode. ]
> [pid  4937] epoll_wait(139,  <unfinished ...>
> [pid  4936] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4909] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4618] epoll_wait(136,  <unfinished ...>
> [pid  4576] futex(0x7ff61ce66474, FUTEX_WAIT_PRIVATE, 1, NULL
<unfinished

> ...>
> [pid  4279] futex(0x7ff61ce62b34, FUTEX_WAIT_PRIVATE, 2203, NULL
> <unfinished ...>
> [pid  4244] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4227] futex(0x7ff56c71ae14, FUTEX_WAIT_PRIVATE, 2237, NULL
> <unfinished ...>
> [pid  4243] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4228] futex(0x7ff5608331a4, FUTEX_WAIT_PRIVATE, 2237, NULL
> <unfinished ...>
> [pid  4208] futex(0x7ff61ce63e54, FUTEX_WAIT_PRIVATE, 5, NULL
<unfinished

> ...>
> [pid  4205] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4204] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4196] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4195] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4194] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4193] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4187] restart_syscall(<... resuming interrupted restart_syscall
...>

> <unfinished ...>
> [pid  4180] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4179] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4177] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4174] accept(133,  <unfinished ...>
> [pid  4173] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4172] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4171] restart_syscall(<... resuming interrupted restart_syscall
...>
> <unfinished ...>
> [pid  4165] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4164] futex(0x7ff61c1f5054, FUTEX_WAIT_PRIVATE, 3, NULL
<unfinished

> ...>
> [pid  4163] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4162] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4161] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4160] futex(0x7ff623d52c20,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff
> <unfinished ...>
> [pid  4159] futex(0x7ff61c1e9d54, FUTEX_WAIT_PRIVATE, 7, NULL
<unfinished
> ...>
> [pid  4158] futex(0x7ff61c1b7f54, FUTEX_WAIT_PRIVATE, 15, NULL
<unfinished
> ...>
> [pid  4157] futex(0x7ff61c1b5554, FUTEX_WAIT_PRIVATE, 19, NULL
<unfinished
> ...>
> [pid  4156] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4155] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4153] futex(0x7ff61c06c754, FUTEX_WAIT_PRIVATE, 7, NULL
<unfinished
> ...>
> [pid  4152] futex(0x7ff61c06ab54, FUTEX_WAIT_PRIVATE, 3, NULL
<unfinished
> ...>
> [pid  4151] futex(0x7ff61c068f54, FUTEX_WAIT_PRIVATE, 7, NULL
<unfinished
> ...>
> [pid  4150] futex(0x7ff61c067354, FUTEX_WAIT_PRIVATE, 7, NULL
<unfinished

> ...>
> [pid  4148] futex(0x7ff61c024a54, FUTEX_WAIT_PRIVATE, 403, NULL
> <unfinished ...>
> [pid  4165] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=849859736}, 0xffffffff <unfinished ...>
> [pid  4147] futex(0x7ff61c022e54, FUTEX_WAIT_PRIVATE, 415, NULL
> <unfinished ...>
> [pid  4146] futex(0x7ff61c021254, FUTEX_WAIT_PRIVATE, 397, NULL
> <unfinished ...>
> [pid  4145] futex(0x7ff61c01f654, FUTEX_WAIT_PRIVATE, 405, NULL
> <unfinished ...>
> [pid  4144] futex(0x7ff61c00e354, FUTEX_WAIT_PRIVATE, 1, NULL
<unfinished
> ...>
> [pid  4136] futex(0x7ff624b729d0, FUTEX_WAIT, 4144, NULL <unfinished
...>

> [pid  4165] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed
> out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=900162344}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=950365105}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=586325}, 0xffffffff) = -1 ETIMEDOUT
(Connection

> timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=50791977}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=100997890}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=151206817}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=201402531}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=251616284}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=301813556}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=352036802}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=402239182}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=452439835}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=502635489}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=552844020}, 0xffffffff <unfinished ...>
> [pid  4156] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> timed out)
> [pid  4156] futex(0x7ff61c1aba28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4156] futex(0x7ff61c1aba54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564858, tv_nsec=506449064}, 0xffffffff <unfinished ...>
> [pid  4165] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed
> out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=603013734}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=653149664}, 0xffffffff^Cstrace: Process 4136
> detached
> strace: Process 4144 detached
> strace: Process 4145 detached
> strace: Process 4146 detached
> strace: Process 4147 detached
> strace: Process 4148 detached
> strace: Process 4150 detached
> strace: Process 4151 detached
> strace: Process 4152 detached
> strace: Process 4153 detached
> ....
>
>
> Could you help us to determine what's wrong with our setup?
>
> Thank you very much,
>
> Kind regards
> Lukas Weiss
>
> ---
> This email has been checked for viruses by AVG.
> https://www.avg.com
>

Reply | Threaded
Open this post in threaded view
|

Re: Re: High CPU usage with Solr 7.7.0

Tomás Fernández Löbbe
Maybe a thread dump would be useful if you still have some instance running
on 7.7

On Wed, Feb 27, 2019 at 7:28 AM Lukas Weiss <[hidden email]>
wrote:

> I can confirm this. Downgrading to 7.6.0 solved the issue.
> Thanks for the hint.
>
>
>
> Von:    "Joe Obernberger" <[hidden email]>
> An:     [hidden email], "Lukas Weiss"
> <[hidden email]>,
> Datum:  27.02.2019 15:59
> Betreff:        Re: High CPU usage with Solr 7.7.0
>
>
>
> Just to add to this.  We upgraded to 7.7.0 and saw very large CPU usage
> on multi core boxes - sustained in the 1200% range.  We then switched to
> 7.6.0 (no other configuration changes) and the problem went away.
>
> We have a 40 node cluster and all 40 nodes had high CPU usage with 3
> indexes stored on HDFS.
>
> -Joe
>
> On 2/27/2019 5:04 AM, Lukas Weiss wrote:
> > Hello,
> >
> > we recently updated our Solr server from 6.6.5 to 7.7.0. Since then, we
> > have problems with the server's CPU usage.
> > We have two Solr cores configured, but even if we clear all indexes and
> do
> > not start the index process, we see 100 CPU usage for both cores.
> >
> > Here's what our top says:
> >
> > root@solr:~ # top
> > top - 09:25:24 up 17:40,  1 user,  load average: 2,28, 2,56, 2,68
> > Threads:  74 total,   3 running,  71 sleeping,   0 stopped,   0 zombie
> > %Cpu0  :100,0 us,  0,0 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,0 si,
> > 0,0 st
> > %Cpu1  :100,0 us,  0,0 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,0 si,
> > 0,0 st
> > %Cpu2  : 11,3 us,  1,0 sy,  0,0 ni, 86,7 id,  0,7 wa,  0,0 hi,  0,3 si,
> > 0,0 st
> > %Cpu3  :  3,0 us,  3,0 sy,  0,0 ni, 93,7 id,  0,3 wa,  0,0 hi,  0,0 si,
> > 0,0 st
> > KiB Mem :  8388608 total,  7859168 free,   496744 used,    32696
> > buff/cache
> > KiB Swap:  2097152 total,  2097152 free,        0 used.  7859168 avail
> Mem
> >
> >
> >    PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+
> COMMAND
> >                P
> > 10209 solr      20   0 6138468 452520  25740 R 99,9  5,4  29:43.45 java
> > -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> > -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 24
> > 10214 solr      20   0 6138468 452520  25740 R 99,9  5,4  28:42.91 java
> > -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> > -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 25
> >
> > The solr server is installed on a Debian Stretch 9.8 (64bit) on Linux
> LXC
> > dedicated Container.
> >
> > Some more server info:
> >
> > root@solr:~ # java -version
> > openjdk version "1.8.0_181"
> > OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
> > OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
> >
> > root@solr:~ # free -m
> >                total        used        free      shared  buff/cache
> > available
> > Mem:           8192         484        7675         701          31 7675
> > Swap:          2048           0        2048
> >
> > We also found something strange if we do an strace of the main process,
> we
> > get lots of ongoing connection timeouts:
> >
> > root@solr:~ # strace -F -p 4136
> > strace: Process 4136 attached with 48 threads
> > strace: [ Process PID=11089 runs in x32 mode. ]
> > [pid  4937] epoll_wait(139,  <unfinished ...>
> > [pid  4936] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4909] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4618] epoll_wait(136,  <unfinished ...>
> > [pid  4576] futex(0x7ff61ce66474, FUTEX_WAIT_PRIVATE, 1, NULL
> <unfinished
> > ...>
> > [pid  4279] futex(0x7ff61ce62b34, FUTEX_WAIT_PRIVATE, 2203, NULL
> > <unfinished ...>
> > [pid  4244] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4227] futex(0x7ff56c71ae14, FUTEX_WAIT_PRIVATE, 2237, NULL
> > <unfinished ...>
> > [pid  4243] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4228] futex(0x7ff5608331a4, FUTEX_WAIT_PRIVATE, 2237, NULL
> > <unfinished ...>
> > [pid  4208] futex(0x7ff61ce63e54, FUTEX_WAIT_PRIVATE, 5, NULL
> <unfinished
> > ...>
> > [pid  4205] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4204] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4196] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4195] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4194] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4193] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4187] restart_syscall(<... resuming interrupted restart_syscall
> ...>
> > <unfinished ...>
> > [pid  4180] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4179] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4177] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4174] accept(133,  <unfinished ...>
> > [pid  4173] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4172] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4171] restart_syscall(<... resuming interrupted restart_syscall
> ...>
> > <unfinished ...>
> > [pid  4165] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4164] futex(0x7ff61c1f5054, FUTEX_WAIT_PRIVATE, 3, NULL
> <unfinished
> > ...>
> > [pid  4163] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4162] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4161] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4160] futex(0x7ff623d52c20,
> > FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff
> > <unfinished ...>
> > [pid  4159] futex(0x7ff61c1e9d54, FUTEX_WAIT_PRIVATE, 7, NULL
> <unfinished
> > ...>
> > [pid  4158] futex(0x7ff61c1b7f54, FUTEX_WAIT_PRIVATE, 15, NULL
> <unfinished
> > ...>
> > [pid  4157] futex(0x7ff61c1b5554, FUTEX_WAIT_PRIVATE, 19, NULL
> <unfinished
> > ...>
> > [pid  4156] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4155] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4153] futex(0x7ff61c06c754, FUTEX_WAIT_PRIVATE, 7, NULL
> <unfinished
> > ...>
> > [pid  4152] futex(0x7ff61c06ab54, FUTEX_WAIT_PRIVATE, 3, NULL
> <unfinished
> > ...>
> > [pid  4151] futex(0x7ff61c068f54, FUTEX_WAIT_PRIVATE, 7, NULL
> <unfinished
> > ...>
> > [pid  4150] futex(0x7ff61c067354, FUTEX_WAIT_PRIVATE, 7, NULL
> <unfinished
> > ...>
> > [pid  4148] futex(0x7ff61c024a54, FUTEX_WAIT_PRIVATE, 403, NULL
> > <unfinished ...>
> > [pid  4165] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> > timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564856, tv_nsec=849859736}, 0xffffffff <unfinished ...>
> > [pid  4147] futex(0x7ff61c022e54, FUTEX_WAIT_PRIVATE, 415, NULL
> > <unfinished ...>
> > [pid  4146] futex(0x7ff61c021254, FUTEX_WAIT_PRIVATE, 397, NULL
> > <unfinished ...>
> > [pid  4145] futex(0x7ff61c01f654, FUTEX_WAIT_PRIVATE, 405, NULL
> > <unfinished ...>
> > [pid  4144] futex(0x7ff61c00e354, FUTEX_WAIT_PRIVATE, 1, NULL
> <unfinished
> > ...>
> > [pid  4136] futex(0x7ff624b729d0, FUTEX_WAIT, 4144, NULL <unfinished
> ...>
> > [pid  4165] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed
> > out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564856, tv_nsec=900162344}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564856, tv_nsec=950365105}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=586325}, 0xffffffff) = -1 ETIMEDOUT
> (Connection
> > timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=50791977}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=100997890}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=151206817}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=201402531}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=251616284}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=301813556}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=352036802}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=402239182}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=452439835}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=502635489}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=552844020}, 0xffffffff <unfinished ...>
> > [pid  4156] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> > timed out)
> > [pid  4156] futex(0x7ff61c1aba28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4156] futex(0x7ff61c1aba54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564858, tv_nsec=506449064}, 0xffffffff <unfinished ...>
> > [pid  4165] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed
> > out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=603013734}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=653149664}, 0xffffffff^Cstrace: Process 4136
> > detached
> > strace: Process 4144 detached
> > strace: Process 4145 detached
> > strace: Process 4146 detached
> > strace: Process 4147 detached
> > strace: Process 4148 detached
> > strace: Process 4150 detached
> > strace: Process 4151 detached
> > strace: Process 4152 detached
> > strace: Process 4153 detached
> > ....
> >
> >
> > Could you help us to determine what's wrong with our setup?
> >
> > Thank you very much,
> >
> > Kind regards
> > Lukas Weiss
> >
> > ---
> > This email has been checked for viruses by AVG.
> > https://www.avg.com
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Antwort: Re: Re: High CPU usage with Solr 7.7.0

Lukas Weiss
This is the information of the Thread Dump screen of the Solr web
interface:

process reaper (8195)
java.util.concurrent.SynchronousQueue$TransferStack@23ec2c53

sun.misc.Unsafe.park​(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos​(LockSupport.java:215)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill​(SynchronousQueue.java:460)
java.util.concurrent.SynchronousQueue$TransferStack.transfer​(SynchronousQueue.java:362)
java.util.concurrent.SynchronousQueue.poll​(SynchronousQueue.java:941)
java.util.concurrent.ThreadPoolExecutor.getTask​(ThreadPoolExecutor.java:1073)
java.util.concurrent.ThreadPoolExecutor.runWorker​(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run​(ThreadPoolExecutor.java:624)
java.lang.Thread.run​(Thread.java:748)
0.8959ms
0.0000ms

commitScheduler-14-thread-35 (8174)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.poll​(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask​(ThreadPoolExecutor.java:1073)
java.util.concurrent.ThreadPoolExecutor.runWorker​(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run​(ThreadPoolExecutor.java:624)
java.lang.Thread.run​(Thread.java:748)
644010.9423ms
643930.0000ms

commitScheduler-16-thread-62 (8173)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.poll​(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask​(ThreadPoolExecutor.java:1073)
java.util.concurrent.ThreadPoolExecutor.runWorker​(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run​(ThreadPoolExecutor.java:624)
java.lang.Thread.run​(Thread.java:748)
644831.4905ms
644740.0000ms

qtp1282287470-8051 (8051)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d2bd65e

sun.misc.Unsafe.park​(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos​(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos​(AbstractQueuedSynchronizer.java:2078)
org.eclipse.jetty.util.BlockingArrayQueue.poll​(BlockingArrayQueue.java:392)
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll​(QueuedThreadPool.java:656)
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800​(QueuedThreadPool.java:46)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run​(QueuedThreadPool.java:720)
java.lang.Thread.run​(Thread.java:748)
14.5521ms
10.0000ms

qtp1282287470-8050 (8050)
sun.nio.ch.EPollArrayWrapper.epollWait​(Native Method)
sun.nio.ch.EPollArrayWrapper.poll​(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect​(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect​(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select​(SelectorImpl.java:97)
sun.nio.ch.SelectorImpl.select​(SelectorImpl.java:101)
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select​(ManagedSelector.java:423)
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce​(ManagedSelector.java:360)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask​(EatWhatYouKill.java:357)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce​(EatWhatYouKill.java:181)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce​(EatWhatYouKill.java:168)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run​(EatWhatYouKill.java:126)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run​(ReservedThreadExecutor.java:366)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob​(QueuedThreadPool.java:765)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run​(QueuedThreadPool.java:683)
java.lang.Thread.run​(Thread.java:748)
10.8397ms
10.0000ms

qtp1282287470-8049 (8049)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4a97a2f0

sun.misc.Unsafe.park​(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos​(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await​(AbstractQueuedSynchronizer.java:2163)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait​(ReservedThreadExecutor.java:292)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run​(ReservedThreadExecutor.java:357)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob​(QueuedThreadPool.java:765)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run​(QueuedThreadPool.java:683)
java.lang.Thread.run​(Thread.java:748)
2.4610ms
0.0000ms

qtp1282287470-8047 (8047)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d2bd65e

sun.misc.Unsafe.park​(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos​(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos​(AbstractQueuedSynchronizer.java:2078)
org.eclipse.jetty.util.BlockingArrayQueue.poll​(BlockingArrayQueue.java:392)
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll​(QueuedThreadPool.java:656)
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800​(QueuedThreadPool.java:46)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run​(QueuedThreadPool.java:720)
java.lang.Thread.run​(Thread.java:748)
8.3763ms
0.0000ms

Scheduler-235237152 (136)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6c16dc65

sun.misc.Unsafe.park​(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos​(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos​(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take​(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take​(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask​(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker​(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run​(ThreadPoolExecutor.java:624)
java.lang.Thread.run​(Thread.java:748)
0.7083ms
0.0000ms

SolrRrdBackendFactory-7-thread-2 (41)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2edafb00

sun.misc.Unsafe.park​(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos​(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos​(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take​(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take​(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask​(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker​(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run​(ThreadPoolExecutor.java:624)
java.lang.Thread.run​(Thread.java:748)
59.4289ms
50.0000ms

autoExpireDocs-19-thread-1 (40)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5db051a3

sun.misc.Unsafe.park​(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos​(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos​(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take​(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take​(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask​(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker​(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run​(ThreadPoolExecutor.java:624)
java.lang.Thread.run​(Thread.java:748)
2600.6087ms
2200.0000ms

autoExpireDocs-18-thread-1 (39)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56962869

sun.misc.Unsafe.park​(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos​(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos​(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take​(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take​(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask​(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker​(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run​(ThreadPoolExecutor.java:624)
java.lang.Thread.run​(Thread.java:748)
3731.4479ms
3730.0000ms

searcherExecutor-10-thread-1 (38)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c39232a

sun.misc.Unsafe.park​(Native Method)
java.util.concurrent.locks.LockSupport.park​(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await​(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take​(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask​(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker​(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run​(ThreadPoolExecutor.java:624)
java.lang.Thread.run​(Thread.java:748)
111.4577ms
110.0000ms

searcherExecutor-11-thread-1 (37)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7542db56

sun.misc.Unsafe.park​(Native Method)
java.util.concurrent.locks.LockSupport.park​(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await​(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take​(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask​(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker​(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run​(ThreadPoolExecutor.java:624)
java.lang.Thread.run​(Thread.java:748)
94.7639ms
90.0000ms

DestroyJavaVM (34)
1409.1001ms
1350.0000ms

Thread-13 (32)
java.lang.Object@2aa0a114

java.lang.Object.wait​(Native Method)
java.lang.Object.wait​(Object.java:502)
org.apache.solr.core.CloserThread.run​(CoreContainer.java:1901)
0.1739ms
0.0000ms

MetricsHistoryHandler-8-thread-1 (29)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4d920149

sun.misc.Unsafe.park​(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos​(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos​(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take​(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take​(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask​(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker​(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run​(ThreadPoolExecutor.java:624)
java.lang.Thread.run​(Thread.java:748)
5333.8818ms
4450.0000ms

SolrRrdBackendFactory-7-thread-1 (28)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2edafb00

sun.misc.Unsafe.park​(Native Method)
java.util.concurrent.locks.LockSupport.park​(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await​(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take​(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take​(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask​(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker​(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run​(ThreadPoolExecutor.java:624)
java.lang.Thread.run​(Thread.java:748)
205.1954ms
90.0000ms

Connection evictor (27)
java.lang.Thread.sleep​(Native Method)
org.apache.http.impl.client.IdleConnectionEvictor$1.run​(IdleConnectionEvictor.java:66)
java.lang.Thread.run​(Thread.java:748)
144.2396ms
110.0000ms

Connection evictor (26)
java.lang.Thread.sleep​(Native Method)
org.apache.http.impl.client.IdleConnectionEvictor$1.run​(IdleConnectionEvictor.java:66)
java.lang.Thread.run​(Thread.java:748)
122.9604ms
120.0000ms

Connection evictor (25)
java.lang.Thread.sleep​(Native Method)
org.apache.http.impl.client.IdleConnectionEvictor$1.run​(IdleConnectionEvictor.java:66)
java.lang.Thread.run​(Thread.java:748)
117.9532ms
110.0000ms

Connection evictor (24)
java.lang.Thread.sleep​(Native Method)
org.apache.http.impl.client.IdleConnectionEvictor$1.run​(IdleConnectionEvictor.java:66)
java.lang.Thread.run​(Thread.java:748)
147.2869ms
120.0000ms

Scheduler-2044366277 (23)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8775ba2

sun.misc.Unsafe.park​(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos​(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos​(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take​(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take​(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask​(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker​(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run​(ThreadPoolExecutor.java:624)
java.lang.Thread.run​(Thread.java:748)
46.1530ms
40.0000ms

qtp1282287470-22 (22)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@439b00fa

sun.misc.Unsafe.park​(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos​(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await​(AbstractQueuedSynchronizer.java:2163)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait​(ReservedThreadExecutor.java:292)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run​(ReservedThreadExecutor.java:357)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob​(QueuedThreadPool.java:765)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run​(QueuedThreadPool.java:683)
java.lang.Thread.run​(Thread.java:748)
59.4703ms
50.0000ms

qtp1282287470-20 (20)
sun.nio.ch.EPollArrayWrapper.epollWait​(Native Method)
sun.nio.ch.EPollArrayWrapper.poll​(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect​(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect​(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select​(SelectorImpl.java:97)
sun.nio.ch.SelectorImpl.select​(SelectorImpl.java:101)
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select​(ManagedSelector.java:423)
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce​(ManagedSelector.java:360)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask​(EatWhatYouKill.java:357)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce​(EatWhatYouKill.java:181)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce​(EatWhatYouKill.java:168)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run​(EatWhatYouKill.java:126)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run​(ReservedThreadExecutor.java:366)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob​(QueuedThreadPool.java:765)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run​(QueuedThreadPool.java:683)
java.lang.Thread.run​(Thread.java:748)
117.0453ms
110.0000ms

qtp1282287470-19 (19)
sun.management.OperatingSystemImpl.getSystemCpuLoad​(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke0​(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke​(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke​(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke​(Method.java:498)
org.apache.solr.util.stats.MetricUtils.addMXBeanMetrics​(MetricUtils.java:587)
org.apache.solr.util.stats.MetricUtils.addMXBeanMetrics​(MetricUtils.java:634)
org.apache.solr.handler.admin.SystemInfoHandler.getSystemInfo​(SystemInfoHandler.java:213)
org.apache.solr.handler.admin.SystemInfoHandler.handleRequestBody​(SystemInfoHandler.java:149)
org.apache.solr.handler.RequestHandlerBase.handleRequest​(RequestHandlerBase.java:199)
org.apache.solr.handler.admin.InfoHandler.handle​(InfoHandler.java:91)
org.apache.solr.handler.admin.InfoHandler.handleRequestBody​(InfoHandler.java:81)
org.apache.solr.handler.RequestHandlerBase.handleRequest​(RequestHandlerBase.java:199)
org.apache.solr.servlet.HttpSolrCall.handleAdmin​(HttpSolrCall.java:735)
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest​(HttpSolrCall.java:716)
org.apache.solr.servlet.HttpSolrCall.call​(HttpSolrCall.java:496)
org.apache.solr.servlet.SolrDispatchFilter.doFilter​(SolrDispatchFilter.java:395)
org.apache.solr.servlet.SolrDispatchFilter.doFilter​(SolrDispatchFilter.java:341)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter​(ServletHandler.java:1602)
org.eclipse.jetty.servlet.ServletHandler.doHandle​(ServletHandler.java:540)
org.eclipse.jetty.server.handler.ScopedHandler.handle​(ScopedHandler.java:146)
org.eclipse.jetty.security.SecurityHandler.handle​(SecurityHandler.java:548)
org.eclipse.jetty.server.handler.HandlerWrapper.handle​(HandlerWrapper.java:132)
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle​(ScopedHandler.java:257)
org.eclipse.jetty.server.session.SessionHandler.doHandle​(SessionHandler.java:1588)
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle​(ScopedHandler.java:255)
org.eclipse.jetty.server.handler.ContextHandler.doHandle​(ContextHandler.java:1345)
org.eclipse.jetty.server.handler.ScopedHandler.nextScope​(ScopedHandler.java:203)
org.eclipse.jetty.servlet.ServletHandler.doScope​(ServletHandler.java:480)
org.eclipse.jetty.server.session.SessionHandler.doScope​(SessionHandler.java:1557)
org.eclipse.jetty.server.handler.ScopedHandler.nextScope​(ScopedHandler.java:201)
org.eclipse.jetty.server.handler.ContextHandler.doScope​(ContextHandler.java:1247)
org.eclipse.jetty.server.handler.ScopedHandler.handle​(ScopedHandler.java:144)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle​(ContextHandlerCollection.java:220)
org.eclipse.jetty.server.handler.HandlerCollection.handle​(HandlerCollection.java:126)
org.eclipse.jetty.server.handler.HandlerWrapper.handle​(HandlerWrapper.java:132)
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle​(RewriteHandler.java:335)
org.eclipse.jetty.server.handler.HandlerWrapper.handle​(HandlerWrapper.java:132)
org.eclipse.jetty.server.Server.handle​(Server.java:502)
org.eclipse.jetty.server.HttpChannel.handle​(HttpChannel.java:364)
org.eclipse.jetty.server.HttpConnection.onFillable​(HttpConnection.java:260)
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded​(AbstractConnection.java:305)
org.eclipse.jetty.io.FillInterest.fillable​(FillInterest.java:103)
org.eclipse.jetty.io.ChannelEndPoint$2.run​(ChannelEndPoint.java:118)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask​(EatWhatYouKill.java:333)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce​(EatWhatYouKill.java:310)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce​(EatWhatYouKill.java:168)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run​(EatWhatYouKill.java:126)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run​(ReservedThreadExecutor.java:366)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob​(QueuedThreadPool.java:765)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run​(QueuedThreadPool.java:683)
java.lang.Thread.run​(Thread.java:748)
58.4642ms
20.0000ms

qtp1282287470-16-acceptor-0@28e7f409-ServerConnector@26200f65{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
(16)
sun.nio.ch.ServerSocketChannelImpl.accept0​(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept​(ServerSocketChannelImpl.java:422)
sun.nio.ch.ServerSocketChannelImpl.accept​(ServerSocketChannelImpl.java:250)
org.eclipse.jetty.server.ServerConnector.accept​(ServerConnector.java:385)
org.eclipse.jetty.server.AbstractConnector$Acceptor.run​(AbstractConnector.java:648)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob​(QueuedThreadPool.java:765)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run​(QueuedThreadPool.java:683)
java.lang.Thread.run​(Thread.java:748)
4.6343ms
0.0000ms

qtp1282287470-15 (15)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d2bd65e

sun.misc.Unsafe.park​(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos​(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos​(AbstractQueuedSynchronizer.java:2078)
org.eclipse.jetty.util.BlockingArrayQueue.poll​(BlockingArrayQueue.java:392)
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll​(QueuedThreadPool.java:656)
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800​(QueuedThreadPool.java:46)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run​(QueuedThreadPool.java:720)
java.lang.Thread.run​(Thread.java:748)
80.6062ms
70.0000ms

qtp1282287470-14 (14)
sun.management.ThreadImpl.getThreadInfo1​(Native Method)
sun.management.ThreadImpl.getThreadInfo​(ThreadImpl.java:176)
org.apache.solr.handler.admin.ThreadDumpHandler.handleRequestBody​(ThreadDumpHandler.java:71)
org.apache.solr.handler.RequestHandlerBase.handleRequest​(RequestHandlerBase.java:199)
org.apache.solr.handler.admin.InfoHandler.handle​(InfoHandler.java:91)
org.apache.solr.handler.admin.InfoHandler.handleRequestBody​(InfoHandler.java:81)
org.apache.solr.handler.RequestHandlerBase.handleRequest​(RequestHandlerBase.java:199)
org.apache.solr.servlet.HttpSolrCall.handleAdmin​(HttpSolrCall.java:735)
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest​(HttpSolrCall.java:716)
org.apache.solr.servlet.HttpSolrCall.call​(HttpSolrCall.java:496)
org.apache.solr.servlet.SolrDispatchFilter.doFilter​(SolrDispatchFilter.java:395)
org.apache.solr.servlet.SolrDispatchFilter.doFilter​(SolrDispatchFilter.java:341)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter​(ServletHandler.java:1602)
org.eclipse.jetty.servlet.ServletHandler.doHandle​(ServletHandler.java:540)
org.eclipse.jetty.server.handler.ScopedHandler.handle​(ScopedHandler.java:146)
org.eclipse.jetty.security.SecurityHandler.handle​(SecurityHandler.java:548)
org.eclipse.jetty.server.handler.HandlerWrapper.handle​(HandlerWrapper.java:132)
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle​(ScopedHandler.java:257)
org.eclipse.jetty.server.session.SessionHandler.doHandle​(SessionHandler.java:1588)
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle​(ScopedHandler.java:255)
org.eclipse.jetty.server.handler.ContextHandler.doHandle​(ContextHandler.java:1345)
org.eclipse.jetty.server.handler.ScopedHandler.nextScope​(ScopedHandler.java:203)
org.eclipse.jetty.servlet.ServletHandler.doScope​(ServletHandler.java:480)
org.eclipse.jetty.server.session.SessionHandler.doScope​(SessionHandler.java:1557)
org.eclipse.jetty.server.handler.ScopedHandler.nextScope​(ScopedHandler.java:201)
org.eclipse.jetty.server.handler.ContextHandler.doScope​(ContextHandler.java:1247)
org.eclipse.jetty.server.handler.ScopedHandler.handle​(ScopedHandler.java:144)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle​(ContextHandlerCollection.java:220)
org.eclipse.jetty.server.handler.HandlerCollection.handle​(HandlerCollection.java:126)
org.eclipse.jetty.server.handler.HandlerWrapper.handle​(HandlerWrapper.java:132)
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle​(RewriteHandler.java:335)
org.eclipse.jetty.server.handler.HandlerWrapper.handle​(HandlerWrapper.java:132)
org.eclipse.jetty.server.Server.handle​(Server.java:502)
org.eclipse.jetty.server.HttpChannel.handle​(HttpChannel.java:364)
org.eclipse.jetty.server.HttpConnection.onFillable​(HttpConnection.java:260)
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded​(AbstractConnection.java:305)
org.eclipse.jetty.io.FillInterest.fillable​(FillInterest.java:103)
org.eclipse.jetty.io.ChannelEndPoint$2.run​(ChannelEndPoint.java:118)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask​(EatWhatYouKill.java:333)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce​(EatWhatYouKill.java:310)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce​(EatWhatYouKill.java:168)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run​(EatWhatYouKill.java:126)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run​(ReservedThreadExecutor.java:366)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob​(QueuedThreadPool.java:765)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run​(QueuedThreadPool.java:683)
java.lang.Thread.run​(Thread.java:748)
75.6621ms
20.0000ms

ShutdownMonitor (12)
java.net.PlainSocketImpl.socketAccept​(Native Method)
java.net.AbstractPlainSocketImpl.accept​(AbstractPlainSocketImpl.java:409)
java.net.ServerSocket.implAccept​(ServerSocket.java:545)
java.net.ServerSocket.accept​(ServerSocket.java:513)
org.eclipse.jetty.server.ShutdownMonitor$ShutdownMonitorRunnable.run​(ShutdownMonitor.java:335)
java.lang.Thread.run​(Thread.java:748)
0.3767ms
0.0000ms
Signal Dispatcher (5)
0.0362ms
0.0000ms

Finalizer (3)
java.lang.ref.ReferenceQueue$Lock@448b0df5

java.lang.Object.wait​(Native Method)
java.lang.ref.ReferenceQueue.remove​(ReferenceQueue.java:144)
java.lang.ref.ReferenceQueue.remove​(ReferenceQueue.java:165)
java.lang.ref.Finalizer$FinalizerThread.run​(Finalizer.java:216)
8.2488ms
0.0000ms

Reference Handler (2)
java.lang.ref.Reference$Lock@19ced464

java.lang.Object.wait​(Native Method)
java.lang.Object.wait​(Object.java:502)
java.lang.ref.Reference.tryHandlePending​(Reference.java:191)
java.lang.ref.Reference$ReferenceHandler.run​(Reference.java:153)



Von:    "Tomás Fernández Löbbe" <[hidden email]>
An:     [hidden email],
Datum:  27.02.2019 19:34
Betreff:        Re: Re: High CPU usage with Solr 7.7.0



Maybe a thread dump would be useful if you still have some instance
running
on 7.7

On Wed, Feb 27, 2019 at 7:28 AM Lukas Weiss <[hidden email]>
wrote:

> I can confirm this. Downgrading to 7.6.0 solved the issue.
> Thanks for the hint.
>
>
>
> Von:    "Joe Obernberger" <[hidden email]>
> An:     [hidden email], "Lukas Weiss"
> <[hidden email]>,
> Datum:  27.02.2019 15:59
> Betreff:        Re: High CPU usage with Solr 7.7.0
>
>
>
> Just to add to this.  We upgraded to 7.7.0 and saw very large CPU usage
> on multi core boxes - sustained in the 1200% range.  We then switched to
> 7.6.0 (no other configuration changes) and the problem went away.
>
> We have a 40 node cluster and all 40 nodes had high CPU usage with 3
> indexes stored on HDFS.
>
> -Joe
>
> On 2/27/2019 5:04 AM, Lukas Weiss wrote:
> > Hello,
> >
> > we recently updated our Solr server from 6.6.5 to 7.7.0. Since then,
we
> > have problems with the server's CPU usage.
> > We have two Solr cores configured, but even if we clear all indexes
and
> do
> > not start the index process, we see 100 CPU usage for both cores.
> >
> > Here's what our top says:
> >
> > root@solr:~ # top
> > top - 09:25:24 up 17:40,  1 user,  load average: 2,28, 2,56, 2,68
> > Threads:  74 total,   3 running,  71 sleeping,   0 stopped,   0 zombie
> > %Cpu0  :100,0 us,  0,0 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,0
si,
> > 0,0 st
> > %Cpu1  :100,0 us,  0,0 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,0
si,
> > 0,0 st
> > %Cpu2  : 11,3 us,  1,0 sy,  0,0 ni, 86,7 id,  0,7 wa,  0,0 hi,  0,3
si,
> > 0,0 st
> > %Cpu3  :  3,0 us,  3,0 sy,  0,0 ni, 93,7 id,  0,3 wa,  0,0 hi,  0,0
si,

> > 0,0 st
> > KiB Mem :  8388608 total,  7859168 free,   496744 used,    32696
> > buff/cache
> > KiB Swap:  2097152 total,  2097152 free,        0 used.  7859168 avail
> Mem
> >
> >
> >    PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+
> COMMAND
> >                P
> > 10209 solr      20   0 6138468 452520  25740 R 99,9  5,4  29:43.45
java
> > -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> > -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 24
> > 10214 solr      20   0 6138468 452520  25740 R 99,9  5,4  28:42.91
java

> > -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> > -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 25
> >
> > The solr server is installed on a Debian Stretch 9.8 (64bit) on Linux
> LXC
> > dedicated Container.
> >
> > Some more server info:
> >
> > root@solr:~ # java -version
> > openjdk version "1.8.0_181"
> > OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
> > OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
> >
> > root@solr:~ # free -m
> >                total        used        free      shared  buff/cache
> > available
> > Mem:           8192         484        7675         701          31
7675
> > Swap:          2048           0        2048
> >
> > We also found something strange if we do an strace of the main
process,

> we
> > get lots of ongoing connection timeouts:
> >
> > root@solr:~ # strace -F -p 4136
> > strace: Process 4136 attached with 48 threads
> > strace: [ Process PID=11089 runs in x32 mode. ]
> > [pid  4937] epoll_wait(139,  <unfinished ...>
> > [pid  4936] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4909] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4618] epoll_wait(136,  <unfinished ...>
> > [pid  4576] futex(0x7ff61ce66474, FUTEX_WAIT_PRIVATE, 1, NULL
> <unfinished
> > ...>
> > [pid  4279] futex(0x7ff61ce62b34, FUTEX_WAIT_PRIVATE, 2203, NULL
> > <unfinished ...>
> > [pid  4244] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4227] futex(0x7ff56c71ae14, FUTEX_WAIT_PRIVATE, 2237, NULL
> > <unfinished ...>
> > [pid  4243] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4228] futex(0x7ff5608331a4, FUTEX_WAIT_PRIVATE, 2237, NULL
> > <unfinished ...>
> > [pid  4208] futex(0x7ff61ce63e54, FUTEX_WAIT_PRIVATE, 5, NULL
> <unfinished
> > ...>
> > [pid  4205] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4204] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4196] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4195] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4194] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4193] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4187] restart_syscall(<... resuming interrupted restart_syscall
> ...>
> > <unfinished ...>
> > [pid  4180] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4179] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4177] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4174] accept(133,  <unfinished ...>
> > [pid  4173] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4172] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4171] restart_syscall(<... resuming interrupted restart_syscall
> ...>
> > <unfinished ...>
> > [pid  4165] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4164] futex(0x7ff61c1f5054, FUTEX_WAIT_PRIVATE, 3, NULL
> <unfinished
> > ...>
> > [pid  4163] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4162] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4161] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4160] futex(0x7ff623d52c20,
> > FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff
> > <unfinished ...>
> > [pid  4159] futex(0x7ff61c1e9d54, FUTEX_WAIT_PRIVATE, 7, NULL
> <unfinished
> > ...>
> > [pid  4158] futex(0x7ff61c1b7f54, FUTEX_WAIT_PRIVATE, 15, NULL
> <unfinished
> > ...>
> > [pid  4157] futex(0x7ff61c1b5554, FUTEX_WAIT_PRIVATE, 19, NULL
> <unfinished
> > ...>
> > [pid  4156] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4155] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  4153] futex(0x7ff61c06c754, FUTEX_WAIT_PRIVATE, 7, NULL
> <unfinished
> > ...>
> > [pid  4152] futex(0x7ff61c06ab54, FUTEX_WAIT_PRIVATE, 3, NULL
> <unfinished
> > ...>
> > [pid  4151] futex(0x7ff61c068f54, FUTEX_WAIT_PRIVATE, 7, NULL
> <unfinished
> > ...>
> > [pid  4150] futex(0x7ff61c067354, FUTEX_WAIT_PRIVATE, 7, NULL
> <unfinished
> > ...>
> > [pid  4148] futex(0x7ff61c024a54, FUTEX_WAIT_PRIVATE, 403, NULL
> > <unfinished ...>
> > [pid  4165] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> > timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564856, tv_nsec=849859736}, 0xffffffff <unfinished ...>
> > [pid  4147] futex(0x7ff61c022e54, FUTEX_WAIT_PRIVATE, 415, NULL
> > <unfinished ...>
> > [pid  4146] futex(0x7ff61c021254, FUTEX_WAIT_PRIVATE, 397, NULL
> > <unfinished ...>
> > [pid  4145] futex(0x7ff61c01f654, FUTEX_WAIT_PRIVATE, 405, NULL
> > <unfinished ...>
> > [pid  4144] futex(0x7ff61c00e354, FUTEX_WAIT_PRIVATE, 1, NULL
> <unfinished
> > ...>
> > [pid  4136] futex(0x7ff624b729d0, FUTEX_WAIT, 4144, NULL <unfinished
> ...>
> > [pid  4165] <... futex resumed> )       = -1 ETIMEDOUT (Connection
timed

> > out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564856, tv_nsec=900162344}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564856, tv_nsec=950365105}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=586325}, 0xffffffff) = -1 ETIMEDOUT
> (Connection
> > timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=50791977}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=100997890}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=151206817}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=201402531}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=251616284}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=301813556}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=352036802}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=402239182}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=452439835}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=502635489}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=552844020}, 0xffffffff <unfinished ...>
> > [pid  4156] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> > timed out)
> > [pid  4156] futex(0x7ff61c1aba28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4156] futex(0x7ff61c1aba54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564858, tv_nsec=506449064}, 0xffffffff <unfinished ...>
> > [pid  4165] <... futex resumed> )       = -1 ETIMEDOUT (Connection
timed

> > out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=603013734}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=653149664}, 0xffffffff^Cstrace: Process 4136
> > detached
> > strace: Process 4144 detached
> > strace: Process 4145 detached
> > strace: Process 4146 detached
> > strace: Process 4147 detached
> > strace: Process 4148 detached
> > strace: Process 4150 detached
> > strace: Process 4151 detached
> > strace: Process 4152 detached
> > strace: Process 4153 detached
> > ....
> >
> >
> > Could you help us to determine what's wrong with our setup?
> >
> > Thank you very much,
> >
> > Kind regards
> > Lukas Weiss
> >
> > ---
> > This email has been checked for viruses by AVG.
> > https://www.avg.com
> >
>
>


Reply | Threaded
Open this post in threaded view
|

Re: High CPU usage with Solr 7.7.0

Lukas Weiss
In reply to this post by Lukas Weiss
I forward this message. Thanks Adam.

Hi,
Apologies, I can’t figure out how to reply to the Solr mailing list.
I just ran across the same high CPU usage issue. I believe it’’s caused by
this commit which was introduced in Solr 7.7.0
https://github.com/apache/lucene-solr/commit/eb652b84edf441d8369f5188cdd5e3ae2b151434#diff-e54b251d166135a1afb7938cfe152bb5
There is a bug in JDK versions <=8 where using 0 threads in the
ScheduledThreadPool causes high CPU usage:
https://bugs.openjdk.java.net/browse/JDK-8129861
Oddly, the latest version
of solr/core/src/java/org/apache/solr/update/CommitTracker.java on
master still uses 0 executors as the default. Presumably most everyone is
using JDK 9 or greater which has the bug fixed, so they don’t experience
the bug.
Feel free to relay this back to the mailing list.
Thanks,
Adam Guthrie





Von:    "Lukas Weiss" <[hidden email]>
An:     [hidden email],
Datum:  27.02.2019 11:13
Betreff:        High CPU usage with Solr 7.7.0



Hello,

we recently updated our Solr server from 6.6.5 to 7.7.0. Since then, we
have problems with the server's CPU usage.
We have two Solr cores configured, but even if we clear all indexes and do

not start the index process, we see 100 CPU usage for both cores.

Here's what our top says:

root@solr:~ # top
top - 09:25:24 up 17:40,  1 user,  load average: 2,28, 2,56, 2,68
Threads:  74 total,   3 running,  71 sleeping,   0 stopped,   0 zombie
%Cpu0  :100,0 us,  0,0 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,0 si,
0,0 st
%Cpu1  :100,0 us,  0,0 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,0 si,
0,0 st
%Cpu2  : 11,3 us,  1,0 sy,  0,0 ni, 86,7 id,  0,7 wa,  0,0 hi,  0,3 si,
0,0 st
%Cpu3  :  3,0 us,  3,0 sy,  0,0 ni, 93,7 id,  0,3 wa,  0,0 hi,  0,0 si,
0,0 st
KiB Mem :  8388608 total,  7859168 free,   496744 used,    32696
buff/cache
KiB Swap:  2097152 total,  2097152 free,        0 used.  7859168 avail Mem



  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND

              P
10209 solr      20   0 6138468 452520  25740 R 99,9  5,4  29:43.45 java
-server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
-XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 24
10214 solr      20   0 6138468 452520  25740 R 99,9  5,4  28:42.91 java
-server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
-XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 25

The solr server is installed on a Debian Stretch 9.8 (64bit) on Linux LXC
dedicated Container.

Some more server info:

root@solr:~ # java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

root@solr:~ # free -m
              total        used        free      shared  buff/cache
available
Mem:           8192         484        7675         701          31 7675
Swap:          2048           0        2048

We also found something strange if we do an strace of the main process, we

get lots of ongoing connection timeouts:

root@solr:~ # strace -F -p 4136
strace: Process 4136 attached with 48 threads
strace: [ Process PID=11089 runs in x32 mode. ]
[pid  4937] epoll_wait(139,  <unfinished ...>
[pid  4936] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4909] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4618] epoll_wait(136,  <unfinished ...>
[pid  4576] futex(0x7ff61ce66474, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
...>
[pid  4279] futex(0x7ff61ce62b34, FUTEX_WAIT_PRIVATE, 2203, NULL
<unfinished ...>
[pid  4244] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4227] futex(0x7ff56c71ae14, FUTEX_WAIT_PRIVATE, 2237, NULL
<unfinished ...>
[pid  4243] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4228] futex(0x7ff5608331a4, FUTEX_WAIT_PRIVATE, 2237, NULL
<unfinished ...>
[pid  4208] futex(0x7ff61ce63e54, FUTEX_WAIT_PRIVATE, 5, NULL <unfinished
...>
[pid  4205] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4204] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4196] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4195] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4194] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4193] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4187] restart_syscall(<... resuming interrupted restart_syscall ...>

<unfinished ...>
[pid  4180] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4179] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4177] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4174] accept(133,  <unfinished ...>
[pid  4173] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4172] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4171] restart_syscall(<... resuming interrupted restart_syscall ...>

<unfinished ...>
[pid  4165] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4164] futex(0x7ff61c1f5054, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished
...>
[pid  4163] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4162] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4161] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4160] futex(0x7ff623d52c20,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff
<unfinished ...>
[pid  4159] futex(0x7ff61c1e9d54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid  4158] futex(0x7ff61c1b7f54, FUTEX_WAIT_PRIVATE, 15, NULL <unfinished

...>
[pid  4157] futex(0x7ff61c1b5554, FUTEX_WAIT_PRIVATE, 19, NULL <unfinished

...>
[pid  4156] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4155] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  4153] futex(0x7ff61c06c754, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid  4152] futex(0x7ff61c06ab54, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished
...>
[pid  4151] futex(0x7ff61c068f54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid  4150] futex(0x7ff61c067354, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid  4148] futex(0x7ff61c024a54, FUTEX_WAIT_PRIVATE, 403, NULL
<unfinished ...>
[pid  4165] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564856, tv_nsec=849859736}, 0xffffffff <unfinished ...>
[pid  4147] futex(0x7ff61c022e54, FUTEX_WAIT_PRIVATE, 415, NULL
<unfinished ...>
[pid  4146] futex(0x7ff61c021254, FUTEX_WAIT_PRIVATE, 397, NULL
<unfinished ...>
[pid  4145] futex(0x7ff61c01f654, FUTEX_WAIT_PRIVATE, 405, NULL
<unfinished ...>
[pid  4144] futex(0x7ff61c00e354, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
...>
[pid  4136] futex(0x7ff624b729d0, FUTEX_WAIT, 4144, NULL <unfinished ...>
[pid  4165] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed
out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564856, tv_nsec=900162344}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564856, tv_nsec=950365105}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=586325}, 0xffffffff) = -1 ETIMEDOUT (Connection
timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=50791977}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=100997890}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=151206817}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=201402531}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=251616284}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=301813556}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=352036802}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=402239182}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=452439835}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=502635489}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=552844020}, 0xffffffff <unfinished ...>
[pid  4156] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
timed out)
[pid  4156] futex(0x7ff61c1aba28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4156] futex(0x7ff61c1aba54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564858, tv_nsec=506449064}, 0xffffffff <unfinished ...>
[pid  4165] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed
out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=603013734}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=653149664}, 0xffffffff^Cstrace: Process 4136
detached
strace: Process 4144 detached
strace: Process 4145 detached
strace: Process 4146 detached
strace: Process 4147 detached
strace: Process 4148 detached
strace: Process 4150 detached
strace: Process 4151 detached
strace: Process 4152 detached
strace: Process 4153 detached
....


Could you help us to determine what's wrong with our setup?

Thank you very much,

Kind regards
Lukas Weiss

Reply | Threaded
Open this post in threaded view
|

Re: High CPU usage with Solr 7.7.0

Erick Erickson
Thanks all. I pushed changes last night, this should be fixed in 7.7.2, 8.1 and master.

Meanwhile, this is a trivial change to one line, so two ways to get by would be

1> just make the change yourself locally. Building Solr from scratch is actually not hard. The “ant package” target will get you the same thing you’d get from downloading the distribution.

2> use Java 9 or greater.

Best,
Erick

> On Mar 25, 2019, at 1:58 AM, Lukas Weiss <[hidden email]> wrote:
>
> I forward this message. Thanks Adam.
>
> Hi,
> Apologies, I can’t figure out how to reply to the Solr mailing list.
> I just ran across the same high CPU usage issue. I believe it’’s caused by
> this commit which was introduced in Solr 7.7.0
> https://github.com/apache/lucene-solr/commit/eb652b84edf441d8369f5188cdd5e3ae2b151434#diff-e54b251d166135a1afb7938cfe152bb5
> There is a bug in JDK versions <=8 where using 0 threads in the
> ScheduledThreadPool causes high CPU usage:
> https://bugs.openjdk.java.net/browse/JDK-8129861
> Oddly, the latest version
> of solr/core/src/java/org/apache/solr/update/CommitTracker.java on
> master still uses 0 executors as the default. Presumably most everyone is
> using JDK 9 or greater which has the bug fixed, so they don’t experience
> the bug.
> Feel free to relay this back to the mailing list.
> Thanks,
> Adam Guthrie
>
>
>
>
>
> Von:    "Lukas Weiss" <[hidden email]>
> An:     [hidden email],
> Datum:  27.02.2019 11:13
> Betreff:        High CPU usage with Solr 7.7.0
>
>
>
> Hello,
>
> we recently updated our Solr server from 6.6.5 to 7.7.0. Since then, we
> have problems with the server's CPU usage.
> We have two Solr cores configured, but even if we clear all indexes and do
>
> not start the index process, we see 100 CPU usage for both cores.
>
> Here's what our top says:
>
> root@solr:~ # top
> top - 09:25:24 up 17:40,  1 user,  load average: 2,28, 2,56, 2,68
> Threads:  74 total,   3 running,  71 sleeping,   0 stopped,   0 zombie
> %Cpu0  :100,0 us,  0,0 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,0 si,
> 0,0 st
> %Cpu1  :100,0 us,  0,0 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,0 si,
> 0,0 st
> %Cpu2  : 11,3 us,  1,0 sy,  0,0 ni, 86,7 id,  0,7 wa,  0,0 hi,  0,3 si,
> 0,0 st
> %Cpu3  :  3,0 us,  3,0 sy,  0,0 ni, 93,7 id,  0,3 wa,  0,0 hi,  0,0 si,
> 0,0 st
> KiB Mem :  8388608 total,  7859168 free,   496744 used,    32696
> buff/cache
> KiB Swap:  2097152 total,  2097152 free,        0 used.  7859168 avail Mem
>
>
>
>  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
>
>              P
> 10209 solr      20   0 6138468 452520  25740 R 99,9  5,4  29:43.45 java
> -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 24
> 10214 solr      20   0 6138468 452520  25740 R 99,9  5,4  28:42.91 java
> -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 25
>
> The solr server is installed on a Debian Stretch 9.8 (64bit) on Linux LXC
> dedicated Container.
>
> Some more server info:
>
> root@solr:~ # java -version
> openjdk version "1.8.0_181"
> OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
> OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
>
> root@solr:~ # free -m
>              total        used        free      shared  buff/cache
> available
> Mem:           8192         484        7675         701          31 7675
> Swap:          2048           0        2048
>
> We also found something strange if we do an strace of the main process, we
>
> get lots of ongoing connection timeouts:
>
> root@solr:~ # strace -F -p 4136
> strace: Process 4136 attached with 48 threads
> strace: [ Process PID=11089 runs in x32 mode. ]
> [pid  4937] epoll_wait(139,  <unfinished ...>
> [pid  4936] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4909] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4618] epoll_wait(136,  <unfinished ...>
> [pid  4576] futex(0x7ff61ce66474, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
> ...>
> [pid  4279] futex(0x7ff61ce62b34, FUTEX_WAIT_PRIVATE, 2203, NULL
> <unfinished ...>
> [pid  4244] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4227] futex(0x7ff56c71ae14, FUTEX_WAIT_PRIVATE, 2237, NULL
> <unfinished ...>
> [pid  4243] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4228] futex(0x7ff5608331a4, FUTEX_WAIT_PRIVATE, 2237, NULL
> <unfinished ...>
> [pid  4208] futex(0x7ff61ce63e54, FUTEX_WAIT_PRIVATE, 5, NULL <unfinished
> ...>
> [pid  4205] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4204] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4196] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4195] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4194] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4193] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4187] restart_syscall(<... resuming interrupted restart_syscall ...>
>
> <unfinished ...>
> [pid  4180] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4179] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4177] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4174] accept(133,  <unfinished ...>
> [pid  4173] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4172] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4171] restart_syscall(<... resuming interrupted restart_syscall ...>
>
> <unfinished ...>
> [pid  4165] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4164] futex(0x7ff61c1f5054, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished
> ...>
> [pid  4163] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4162] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4161] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4160] futex(0x7ff623d52c20,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff
> <unfinished ...>
> [pid  4159] futex(0x7ff61c1e9d54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
> ...>
> [pid  4158] futex(0x7ff61c1b7f54, FUTEX_WAIT_PRIVATE, 15, NULL <unfinished
>
> ...>
> [pid  4157] futex(0x7ff61c1b5554, FUTEX_WAIT_PRIVATE, 19, NULL <unfinished
>
> ...>
> [pid  4156] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4155] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  4153] futex(0x7ff61c06c754, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
> ...>
> [pid  4152] futex(0x7ff61c06ab54, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished
> ...>
> [pid  4151] futex(0x7ff61c068f54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
> ...>
> [pid  4150] futex(0x7ff61c067354, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
> ...>
> [pid  4148] futex(0x7ff61c024a54, FUTEX_WAIT_PRIVATE, 403, NULL
> <unfinished ...>
> [pid  4165] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=849859736}, 0xffffffff <unfinished ...>
> [pid  4147] futex(0x7ff61c022e54, FUTEX_WAIT_PRIVATE, 415, NULL
> <unfinished ...>
> [pid  4146] futex(0x7ff61c021254, FUTEX_WAIT_PRIVATE, 397, NULL
> <unfinished ...>
> [pid  4145] futex(0x7ff61c01f654, FUTEX_WAIT_PRIVATE, 405, NULL
> <unfinished ...>
> [pid  4144] futex(0x7ff61c00e354, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
> ...>
> [pid  4136] futex(0x7ff624b729d0, FUTEX_WAIT, 4144, NULL <unfinished ...>
> [pid  4165] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed
> out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=900162344}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=950365105}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=586325}, 0xffffffff) = -1 ETIMEDOUT (Connection
> timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=50791977}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=100997890}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=151206817}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=201402531}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=251616284}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=301813556}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=352036802}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=402239182}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=452439835}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=502635489}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=552844020}, 0xffffffff <unfinished ...>
> [pid  4156] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> timed out)
> [pid  4156] futex(0x7ff61c1aba28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4156] futex(0x7ff61c1aba54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564858, tv_nsec=506449064}, 0xffffffff <unfinished ...>
> [pid  4165] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed
> out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=603013734}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid  4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=653149664}, 0xffffffff^Cstrace: Process 4136
> detached
> strace: Process 4144 detached
> strace: Process 4145 detached
> strace: Process 4146 detached
> strace: Process 4147 detached
> strace: Process 4148 detached
> strace: Process 4150 detached
> strace: Process 4151 detached
> strace: Process 4152 detached
> strace: Process 4153 detached
> ....
>
>
> Could you help us to determine what's wrong with our setup?
>
> Thank you very much,
>
> Kind regards
> Lukas Weiss
>