Solr 8.2 replicas use only 1 CPU at 100% every solr.autoCommit.maxTime minutes
We run Solr 8.2.0
* with Amazon Corretto 18.104.22.168.1 SDK (java arguments shown in ),
* on Ubuntu 18.04
* on AWS EC2 m5.2xlarge with 8 CPUs and 32GB of RAM
* with -Xmx16g .
We have migrated from Solr 3.5 and in big core (16GB) replicas we have
started to suffer degraded service. The replica’s ReplicationHandler is in
 and the master’s updateHandler in .
We notice every 5 mins (the value for solr.autoCommit.maxTime) the
* Solr uses all 8 CPUs. Suddenly for ~30 sec, it uses only 1 CPU at 100%
and the rest of the CPUs are idle (mpstat ). In our previous setup with
Solr 3 we used up to 80% of all CPUs.
* During that time the solr queries suddenly take more than 1 second, up to
30 sec (or more). The same queries otherwise need less than 1 sec to
* The disk does not seem to be a bottleneck (iostat ).
* Memory does not seem to be a bottleneck (vmstat ).
* CPU (apart from the single CPU issue) does not seem to be a bottleneck
(mpstat  & pidstat ).
* We are no java/GC experts but It does not seem to be GC related .
We have tried reducing the heap to 8 and 2GB with no positive effect. We
have tested different autoCommit.maxTime values. Reducing it to 60 seconds
makes things unbearable. 5 minutes is not significantly different than 10.
Do you have any pointers to proceed debugging the issue?
Detailed example problem that repeats every solr.autoCommit.maxTime minutes
on the replicas:
* From 12:36 to 12:39:04 queries are fast to serve . Solr consumes CPU
from all 8 CPUs (mpstat ). The metric solr.jvm.threads.blocked.count is
* From 12:39:04 to 12:39:25 queries are slow to respond . Solr consumes
only 1 out of 8 CPUs, the other 7 CPUs are idle (mpstat ). The metric
solr.jvm.threads.blocked.count grows from 0 to a big 2 digit number .
* After 12:39:25 and until the next poll of a commit things are normal.
Some additional information. We noticed (through the admin's "Thread Dump"
/solr/#/~threads) that whenever we see this behavior the all the threads
that block show the same stacktrace  and block at