Solr 7.6.0 High CPU Usage Deadlock (Generating 70% of futex system call)

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr 7.6.0 High CPU Usage Deadlock (Generating 70% of futex system call)

Daniel de Oliveira Mantovani
I have a Solr 7.6.0 Cloud with 4 nodes and while indexing Solr hangs.

$ java -version
java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode

$ sudo strace -f -c -p 5972
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 71.02   33.329124         662     50335     13317 futex
 21.43   10.055086      335170        30        28 restart_syscall
  7.49    3.512846       14760       238           epoll_wait
  0.07    0.030526        2544        12           poll
  0.00    0.001571           5       322           write
  0.00    0.000439           4       117           times
  0.00    0.000355           2       183         4 read
  0.00    0.000236           3        72           mprotect
  0.00    0.000161           1       118           epoll_ctl
  0.00    0.000126           0       352           stat
  0.00    0.000089          11         8           getdents
  0.00    0.000077          10         8           sendto
  0.00    0.000069          69         1           statfs
  0.00    0.000043          22         2           writev
  0.00    0.000029           4         7           openat
  0.00    0.000021           4         6           fstat
  0.00    0.000018           1        22           lstat
  0.00    0.000018           9         2           dup
  0.00    0.000010           1         9           close
  0.00    0.000010           3         4           fcntl
  0.00    0.000005           3         2           access
  0.00    0.000002           2         1           mmap
  0.00    0.000002           0        26           sched_yield
  0.00    0.000001           0         8           recvfrom
  0.00    0.000001           1         2           readlink
------ ----------- ----------- --------- --------- ----------------
100.00   46.930865                 51887     13349 total

$ sudo strace -f -p 5972
strace: Process 5972 attached with 86 threads
[pid  7524] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  7523] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  7511] epoll_wait(142,  <unfinished ...>
[pid  7471] epoll_wait(160,  <unfinished ...>
[pid  7266] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  7267] epoll_wait(151,  <unfinished ...>
[pid  7264] futex(0x7fcbd8f3eb98, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid  7265] epoll_wait(157,  <unfinished ...>
[pid  6311] futex(0x7fcabc8c42f8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid  6310] futex(0x7fcab843c998, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid  6281] epoll_wait(154,  <unfinished ...>
[pid  6280] futex(0x7fca68001d7c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid  6278] futex(0x7fca74001d78, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid  6277] accept(136,  <unfinished ...>
[pid  6272] futex(0x7fca7c001d7c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid  6271] epoll_wait(208,  <unfinished ...>
[pid  6270] epoll_wait(205,  <unfinished ...>
[pid  6262] epoll_wait(199,  <unfinished ...>
[pid  6261] epoll_wait(196,  <unfinished ...>
[pid  6258] epoll_wait(193,  <unfinished ...>
[pid  6269] epoll_wait(202,  <unfinished ...>
[pid  6257] epoll_wait(190,  <unfinished ...>
[pid  6256] epoll_wait(187,  <unfinished ...>
[pid  6255] epoll_wait(184,  <unfinished ...>
[pid  6253] epoll_wait(178,  <unfinished ...>
[pid  6250] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  6252] epoll_wait(175,  <unfinished ...>
[pid  6251] epoll_wait(172,  <unfinished ...>
[pid  6254] epoll_wait(181,  <unfinished ...>
[pid  6246] futex(0x7fcbd8f3f378, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid  6238] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  6237] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  6235] futex(0x7fcbd83d97b8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid  6234] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  6233] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  6232] futex(0x7fcad8002608, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid  6231] futex(0x7fcbd89425ac, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid  6049] futex(0x7fcaf4002388, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid  6230] epoll_wait(134,  <unfinished ...>
[pid  6048] futex(0x7fcbd8db9ac8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid  6047] epoll_wait(130,  <unfinished ...>
[pid  6046] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid  6045] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>

JVM args:


   - -DSTOP.KEY=solrrocks-DSTOP.PORT=7983-Djetty.home=/opt/solr/server-Djetty.port=8983-Dlog4j.configurationFile=file:/opt/solr/server/resources/log4j2.xml-Dsolr.data.home=-Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf-Dsolr.install.dir=/opt/solr-Dsolr.jetty.https.port=8983-Dsolr.log.dir=/opt/solr/example/cloud/node1/solr/../logs-Dsolr.log.muteconsole-Dsolr.solr.home=/opt/solr/example/cloud/node1/solr-Duser.timezone=UTC-DzkClientTimeout=15000-DzkHost=localhost:2181-XX:+CMSParallelRemarkEnabled-XX:+CMSScavengeBeforeRemark-XX:+ParallelRefProcEnabled-XX:+PrintGCApplicationStoppedTime-XX:+PrintGCDateStamps-XX:+PrintGCDetails-XX:+PrintGCTimeStamps-XX:+PrintHeapAtGC-XX:+PrintTenuringDistribution-XX:+UseCMSInitiatingOccupancyOnly-XX:+UseConcMarkSweepGC-XX:+UseGCLogFileRotation-XX:-OmitStackTraceInFastThrow-XX:CMSInitiatingOccupancyFraction=50-XX:CMSMaxAbortablePrecleanTime=6000-XX:ConcGCThreads=4-XX:GCLogFileSize=20M-XX:MaxTenuringThreshold=8-XX:NewRatio=3-XX:NumberOfGCLogFiles=9-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh
   8983
   /opt/solr/example/cloud/node1/solr/../logs-XX:ParallelGCThreads=4-XX:PretenureSizeThreshold=64m-XX:SurvivorRatio=4-XX:TargetSurvivorRatio=90-Xloggc:/opt/solr/example/cloud/node1/solr/../logs/solr_gc.log-Xms512m-Xmx15g-Xss256k-verbose:gc

Java Thread Dumb:
name
cpuTime / userTime
process reaper (135)
java.util.concurrent.SynchronousQueue$TransferStack@70b8631c
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
• java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill
(SynchronousQueue.java:460)
• java.util.concurrent.SynchronousQueue$TransferStack.transfer
(SynchronousQueue.java:362)
• java.util.concurrent.SynchronousQueue.poll (SynchronousQueue.java:941)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1073)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
17.2219ms
0.0000ms
qtp1571967156-129 (129)
36.0159ms
0.0000ms
process reaper (117)
java.util.concurrent.SynchronousQueue$TransferStack@70b8631c
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
• java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill
(SynchronousQueue.java:460)
• java.util.concurrent.SynchronousQueue$TransferStack.transfer
(SynchronousQueue.java:362)
• java.util.concurrent.SynchronousQueue.poll (SynchronousQueue.java:941)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1073)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
19.2751ms
0.0000ms
qtp1571967156-116 (116)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
1.5766ms
0.0000ms
qtp1571967156-115 (115)
• sun.management.ThreadImpl.getThreadInfo1 (Native Method)
• sun.management.ThreadImpl.getThreadInfo (ThreadImpl.java:176)
• org.apache.solr.handler.admin.ThreadDumpHandler.handleRequestBody
(ThreadDumpHandler.java:71)
• org.apache.solr.handler.RequestHandlerBase.handleRequest
(RequestHandlerBase.java:199)
• org.apache.solr.handler.admin.InfoHandler.handle (InfoHandler.java:91)
• org.apache.solr.handler.admin.InfoHandler.handleRequestBody
(InfoHandler.java:81)
• org.apache.solr.handler.RequestHandlerBase.handleRequest
(RequestHandlerBase.java:199)
• org.apache.solr.servlet.HttpSolrCall.handleAdmin (HttpSolrCall.java:734)
• org.apache.solr.servlet.HttpSolrCall.handleAdminRequest
(HttpSolrCall.java:715)
• org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:496)
• org.apache.solr.servlet.SolrDispatchFilter.doFilter
(SolrDispatchFilter.java:377)
• org.apache.solr.servlet.SolrDispatchFilter.doFilter
(SolrDispatchFilter.java:323)
• org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter
(ServletHandler.java:1634)
• org.eclipse.jetty.servlet.ServletHandler.doHandle
(ServletHandler.java:533)
• org.eclipse.jetty.server.handler.ScopedHandler.handle
(ScopedHandler.java:146)
• org.eclipse.jetty.security.SecurityHandler.handle
(SecurityHandler.java:548)
• org.eclipse.jetty.server.handler.HandlerWrapper.handle
(HandlerWrapper.java:132)
• org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
(ScopedHandler.java:257)
• org.eclipse.jetty.server.session.SessionHandler.doHandle
(SessionHandler.java:1595)
• org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
(ScopedHandler.java:255)
• org.eclipse.jetty.server.handler.ContextHandler.doHandle
(ContextHandler.java:1317)
• org.eclipse.jetty.server.handler.ScopedHandler.nextScope
(ScopedHandler.java:203)
• org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:473)
• org.eclipse.jetty.server.session.SessionHandler.doScope
(SessionHandler.java:1564)
• org.eclipse.jetty.server.handler.ScopedHandler.nextScope
(ScopedHandler.java:201)
• org.eclipse.jetty.server.handler.ContextHandler.doScope
(ContextHandler.java:1219)
• org.eclipse.jetty.server.handler.ScopedHandler.handle
(ScopedHandler.java:144)
• org.eclipse.jetty.server.handler.ContextHandlerCollection.handle
(ContextHandlerCollection.java:219)
• org.eclipse.jetty.server.handler.HandlerCollection.handle
(HandlerCollection.java:126)
• org.eclipse.jetty.server.handler.HandlerWrapper.handle
(HandlerWrapper.java:132)
• org.eclipse.jetty.rewrite.handler.RewriteHandler.handle
(RewriteHandler.java:335)
• org.eclipse.jetty.server.handler.HandlerWrapper.handle
(HandlerWrapper.java:132)
• org.eclipse.jetty.server.Server.handle (Server.java:531)
• org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:352)
• org.eclipse.jetty.server.HttpConnection.onFillable
(HttpConnection.java:260)
• org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded
(AbstractConnection.java:281)
• org.eclipse.jetty.io.FillInterest.fillable (FillInterest.java:102)
• org.eclipse.jetty.io.ChannelEndPoint$2.run (ChannelEndPoint.java:118)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask
(EatWhatYouKill.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:310)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
81.0773ms
70.0000ms
qtp1571967156-114 (114)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
14.7759ms
10.0000ms
qtp1571967156-113 (113)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
1.7541ms
0.0000ms
qtp1571967156-112 (112)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
28.2561ms
10.0000ms
qtp1571967156-109 (109)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
20.5324ms
10.0000ms
qtp1571967156-107 (107)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
13.4180ms
0.0000ms
qtp1571967156-90 (90)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
108.5163ms
100.0000ms
Scheduler-842741472 (89)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@27022346
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
(AbstractQueuedSynchronizer.java:2078)
• java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:1093)
• java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:809)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1074)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
5.4794ms
0.0000ms
qtp1571967156-88 (88)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
38.6417ms
10.0000ms
SolrRrdBackendFactory-11-thread-2 (87)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7813101a
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
(AbstractQueuedSynchronizer.java:2078)
• java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:1093)
• java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:809)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1074)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
3.3288ms
0.0000ms
searcherExecutor-15-thread-1 (83)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4004f6b2
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2039)
• java.util.concurrent.LinkedBlockingQueue.take
(LinkedBlockingQueue.java:442)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1074)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
8.9663ms
0.0000ms
searcherExecutor-14-thread-1 (84)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7397cd2d
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2039)
• java.util.concurrent.LinkedBlockingQueue.take
(LinkedBlockingQueue.java:442)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1074)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
10.0677ms
0.0000ms
qtp1571967156-79 (79)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
110.6471ms
90.0000ms
qtp1571967156-78-acceptor-3@6942c593-ServerConnector
@362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (78)
java.lang.Object@4539d80a
• sun.nio.ch.ServerSocketChannelImpl.accept
(ServerSocketChannelImpl.java:234)
• org.eclipse.jetty.server.ServerConnector.accept (ServerConnector.java:369)
• org.eclipse.jetty.server.AbstractConnector$Acceptor.run
(AbstractConnector.java:639)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
4.9023ms
0.0000ms
qtp1571967156-77-acceptor-2@26493f4e-ServerConnector
@362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (77)
java.lang.Object@4539d80a
• sun.nio.ch.ServerSocketChannelImpl.accept
(ServerSocketChannelImpl.java:234)
• org.eclipse.jetty.server.ServerConnector.accept (ServerConnector.java:369)
• org.eclipse.jetty.server.AbstractConnector$Acceptor.run
(AbstractConnector.java:639)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
5.0835ms
0.0000ms
qtp1571967156-76-acceptor-1@292b8c3e-ServerConnector
@362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (76)
• sun.nio.ch.ServerSocketChannelImpl.accept0 (Native Method)
• sun.nio.ch.ServerSocketChannelImpl.accept
(ServerSocketChannelImpl.java:422)
• sun.nio.ch.ServerSocketChannelImpl.accept
(ServerSocketChannelImpl.java:250)
• org.eclipse.jetty.server.ServerConnector.accept (ServerConnector.java:369)
• org.eclipse.jetty.server.AbstractConnector$Acceptor.run
(AbstractConnector.java:639)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
5.6738ms
0.0000ms
qtp1571967156-75-acceptor-0@10c4f96f-ServerConnector
@362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (75)
java.lang.Object@4539d80a
• sun.nio.ch.ServerSocketChannelImpl.accept
(ServerSocketChannelImpl.java:234)
• org.eclipse.jetty.server.ServerConnector.accept (ServerConnector.java:369)
• org.eclipse.jetty.server.AbstractConnector$Acceptor.run
(AbstractConnector.java:639)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
4.9762ms
0.0000ms
qtp1571967156-74 (74)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
(AbstractQueuedSynchronizer.java:2078)
• org.eclipse.jetty.util.BlockingArrayQueue.poll
(BlockingArrayQueue.java:392)
• org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
(QueuedThreadPool.java:653)
• org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
(QueuedThreadPool.java:48)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:717)
• java.lang.Thread.run (Thread.java:748)
4.6754ms
0.0000ms
qtp1571967156-73 (73)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
(AbstractQueuedSynchronizer.java:2078)
• org.eclipse.jetty.util.BlockingArrayQueue.poll
(BlockingArrayQueue.java:392)
• org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
(QueuedThreadPool.java:653)
• org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
(QueuedThreadPool.java:48)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:717)
• java.lang.Thread.run (Thread.java:748)
6.0771ms
0.0000ms
qtp1571967156-72 (72)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
28.1516ms
10.0000ms
qtp1571967156-71 (71)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
31.7190ms
10.0000ms
qtp1571967156-70 (70)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
22.8869ms
0.0000ms
qtp1571967156-69 (69)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
(AbstractQueuedSynchronizer.java:2078)
• org.eclipse.jetty.util.BlockingArrayQueue.poll
(BlockingArrayQueue.java:392)
• org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
(QueuedThreadPool.java:653)
• org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
(QueuedThreadPool.java:48)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:717)
• java.lang.Thread.run (Thread.java:748)
5.6719ms
0.0000ms
qtp1571967156-68 (68)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
7.0843ms
0.0000ms
qtp1571967156-67 (67)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
25.2077ms
10.0000ms
qtp1571967156-66 (66)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
59.3490ms
40.0000ms
qtp1571967156-65 (65)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
77.9195ms
60.0000ms
qtp1571967156-64 (64)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
44.4205ms
30.0000ms
qtp1571967156-63 (63)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
129.4624ms
90.0000ms
qtp1571967156-62 (62)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@421090bf
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2163)

org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait
(ReservedThreadExecutor.java:292)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:357)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
51.2605ms
30.0000ms
DestroyJavaVM (61)

2002.8222ms
1850.0000ms
Thread-13 (58)
java.lang.Object@f2f13c1
• java.lang.Object.wait (Native Method)
• java.lang.Object.wait (Object.java:502)
• org.apache.solr.core.CloserThread.run (CoreContainer.java:1850)
0.7894ms
0.0000ms
MetricsHistoryHandler-12-thread-1 (55)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@45d73a6
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
(AbstractQueuedSynchronizer.java:2078)
• java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:1093)
• java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:809)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1074)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
348.4029ms
310.0000ms
SolrRrdBackendFactory-11-thread-1 (54)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7813101a
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2039)
• java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:1088)
• java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:809)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1074)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
5.7281ms
0.0000ms
OverseerAutoScalingTriggerThread-72060990031069185-192.168.1.40:8983_solr-n_0000001018
(52)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@75fd9961
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2039)
• org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run
(OverseerTriggerThread.java:189)
• java.lang.Thread.run (Thread.java:748)
58.1405ms
50.0000ms
OverseerCollectionConfigSetProcessor-72060990031069185-192.168.1.40:8983_solr-n_0000001018
(51)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@55c10b44
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
(AbstractQueuedSynchronizer.java:2078)
• org.apache.solr.cloud.ZkDistributedQueue.peekElements
(ZkDistributedQueue.java:463)
• org.apache.solr.cloud.OverseerTaskQueue.peekTopN
(OverseerTaskQueue.java:243)
• org.apache.solr.cloud.OverseerTaskProcessor.run
(OverseerTaskProcessor.java:233)
• java.lang.Thread.run (Thread.java:748)
220.6397ms
170.0000ms
OverseerStateUpdate-72060990031069185-192.168.1.40:8983_solr-n_0000001018
(47)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2287ab26
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
(AbstractQueuedSynchronizer.java:2078)
• org.apache.solr.cloud.ZkDistributedQueue.peekElements
(ZkDistributedQueue.java:463)
• org.apache.solr.cloud.Overseer$ClusterStateUpdater.run (Overseer.java:207)
• java.lang.Thread.run (Thread.java:748)
189.5649ms
140.0000ms
zkConnectionManagerCallback-13-thread-1 (50)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4da83c93
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2039)
• java.util.concurrent.LinkedBlockingQueue.take
(LinkedBlockingQueue.java:442)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1074)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
1.1550ms
0.0000ms
main-EventThread (49)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@540792d6
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2039)
• java.util.concurrent.LinkedBlockingQueue.take
(LinkedBlockingQueue.java:442)
• org.apache.zookeeper.ClientCnxn$EventThread.run (ClientCnxn.java:502)
2.6375ms
0.0000ms
main-SendThread(localhost:2181) (48)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• org.apache.zookeeper.ClientCnxnSocketNIO.doTransport
(ClientCnxnSocketNIO.java:349)
• org.apache.zookeeper.ClientCnxn$SendThread.run (ClientCnxn.java:1144)
52.9289ms
10.0000ms
zkConnectionManagerCallback-6-thread-1 (44)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@9c13839
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2039)
• java.util.concurrent.LinkedBlockingQueue.take
(LinkedBlockingQueue.java:442)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1074)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
1.1128ms
0.0000ms
main-EventThread (43)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16806cbc
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2039)
• java.util.concurrent.LinkedBlockingQueue.take
(LinkedBlockingQueue.java:442)
• org.apache.zookeeper.ClientCnxn$EventThread.run (ClientCnxn.java:502)


--

--
Daniel Mantovani
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.6.0 High CPU Usage Deadlock (Generating 70% of futex system call)

Shalin Shekhar Mangar
Perhaps this bug in the kernel?
https://groups.google.com/forum/#!topic/mechanical-sympathy/QbmpZxp6C64

Can you check if you are running an affected kernel?

On Fri, Nov 1, 2019 at 7:09 AM Daniel de Oliveira Mantovani <
[hidden email]> wrote:

> I have a Solr 7.6.0 Cloud with 4 nodes and while indexing Solr hangs.
>
> $ java -version
> java version "1.8.0_231"
> Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
> Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode
>
> $ sudo strace -f -c -p 5972
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
>  71.02   33.329124         662     50335     13317 futex
>  21.43   10.055086      335170        30        28 restart_syscall
>   7.49    3.512846       14760       238           epoll_wait
>   0.07    0.030526        2544        12           poll
>   0.00    0.001571           5       322           write
>   0.00    0.000439           4       117           times
>   0.00    0.000355           2       183         4 read
>   0.00    0.000236           3        72           mprotect
>   0.00    0.000161           1       118           epoll_ctl
>   0.00    0.000126           0       352           stat
>   0.00    0.000089          11         8           getdents
>   0.00    0.000077          10         8           sendto
>   0.00    0.000069          69         1           statfs
>   0.00    0.000043          22         2           writev
>   0.00    0.000029           4         7           openat
>   0.00    0.000021           4         6           fstat
>   0.00    0.000018           1        22           lstat
>   0.00    0.000018           9         2           dup
>   0.00    0.000010           1         9           close
>   0.00    0.000010           3         4           fcntl
>   0.00    0.000005           3         2           access
>   0.00    0.000002           2         1           mmap
>   0.00    0.000002           0        26           sched_yield
>   0.00    0.000001           0         8           recvfrom
>   0.00    0.000001           1         2           readlink
> ------ ----------- ----------- --------- --------- ----------------
> 100.00   46.930865                 51887     13349 total
>
> $ sudo strace -f -p 5972
> strace: Process 5972 attached with 86 threads
> [pid  7524] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  7523] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  7511] epoll_wait(142,  <unfinished ...>
> [pid  7471] epoll_wait(160,  <unfinished ...>
> [pid  7266] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  7267] epoll_wait(151,  <unfinished ...>
> [pid  7264] futex(0x7fcbd8f3eb98, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid  7265] epoll_wait(157,  <unfinished ...>
> [pid  6311] futex(0x7fcabc8c42f8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid  6310] futex(0x7fcab843c998, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid  6281] epoll_wait(154,  <unfinished ...>
> [pid  6280] futex(0x7fca68001d7c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid  6278] futex(0x7fca74001d78, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid  6277] accept(136,  <unfinished ...>
> [pid  6272] futex(0x7fca7c001d7c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid  6271] epoll_wait(208,  <unfinished ...>
> [pid  6270] epoll_wait(205,  <unfinished ...>
> [pid  6262] epoll_wait(199,  <unfinished ...>
> [pid  6261] epoll_wait(196,  <unfinished ...>
> [pid  6258] epoll_wait(193,  <unfinished ...>
> [pid  6269] epoll_wait(202,  <unfinished ...>
> [pid  6257] epoll_wait(190,  <unfinished ...>
> [pid  6256] epoll_wait(187,  <unfinished ...>
> [pid  6255] epoll_wait(184,  <unfinished ...>
> [pid  6253] epoll_wait(178,  <unfinished ...>
> [pid  6250] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  6252] epoll_wait(175,  <unfinished ...>
> [pid  6251] epoll_wait(172,  <unfinished ...>
> [pid  6254] epoll_wait(181,  <unfinished ...>
> [pid  6246] futex(0x7fcbd8f3f378, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid  6238] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  6237] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  6235] futex(0x7fcbd83d97b8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid  6234] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  6233] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  6232] futex(0x7fcad8002608, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid  6231] futex(0x7fcbd89425ac, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid  6049] futex(0x7fcaf4002388, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid  6230] epoll_wait(134,  <unfinished ...>
> [pid  6048] futex(0x7fcbd8db9ac8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid  6047] epoll_wait(130,  <unfinished ...>
> [pid  6046] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid  6045] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
>
> JVM args:
>
>
>    -
> -DSTOP.KEY=solrrocks-DSTOP.PORT=7983-Djetty.home=/opt/solr/server-Djetty.port=8983-Dlog4j.configurationFile=file:/opt/solr/server/resources/log4j2.xml-Dsolr.data.home=-Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf-Dsolr.install.dir=/opt/solr-Dsolr.jetty.https.port=8983-Dsolr.log.dir=/opt/solr/example/cloud/node1/solr/../logs-Dsolr.log.muteconsole-Dsolr.solr.home=/opt/solr/example/cloud/node1/solr-Duser.timezone=UTC-DzkClientTimeout=15000-DzkHost=localhost:2181-XX:+CMSParallelRemarkEnabled-XX:+CMSScavengeBeforeRemark-XX:+ParallelRefProcEnabled-XX:+PrintGCApplicationStoppedTime-XX:+PrintGCDateStamps-XX:+PrintGCDetails-XX:+PrintGCTimeStamps-XX:+PrintHeapAtGC-XX:+PrintTenuringDistribution-XX:+UseCMSInitiatingOccupancyOnly-XX:+UseConcMarkSweepGC-XX:+UseGCLogFileRotation-XX:-OmitStackTraceInFastThrow-XX:CMSInitiatingOccupancyFraction=50-XX:CMSMaxAbortablePrecleanTime=6000-XX:ConcGCThreads=4-XX:GCLogFileSize=20M-XX:MaxTenuringThreshold=8-XX:NewRatio=3-XX:NumberOfGCLogFiles=9-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh
>    8983
>
>  /opt/solr/example/cloud/node1/solr/../logs-XX:ParallelGCThreads=4-XX:PretenureSizeThreshold=64m-XX:SurvivorRatio=4-XX:TargetSurvivorRatio=90-Xloggc:/opt/solr/example/cloud/node1/solr/../logs/solr_gc.log-Xms512m-Xmx15g-Xss256k-verbose:gc
>
> Java Thread Dumb:
> name
> cpuTime / userTime
> process reaper (135)
> java.util.concurrent.SynchronousQueue$TransferStack@70b8631c
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> • java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill
> (SynchronousQueue.java:460)
> • java.util.concurrent.SynchronousQueue$TransferStack.transfer
> (SynchronousQueue.java:362)
> • java.util.concurrent.SynchronousQueue.poll (SynchronousQueue.java:941)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1073)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 17.2219ms
> 0.0000ms
> qtp1571967156-129 (129)
> 36.0159ms
> 0.0000ms
> process reaper (117)
> java.util.concurrent.SynchronousQueue$TransferStack@70b8631c
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> • java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill
> (SynchronousQueue.java:460)
> • java.util.concurrent.SynchronousQueue$TransferStack.transfer
> (SynchronousQueue.java:362)
> • java.util.concurrent.SynchronousQueue.poll (SynchronousQueue.java:941)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1073)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 19.2751ms
> 0.0000ms
> qtp1571967156-116 (116)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 1.5766ms
> 0.0000ms
> qtp1571967156-115 (115)
> • sun.management.ThreadImpl.getThreadInfo1 (Native Method)
> • sun.management.ThreadImpl.getThreadInfo (ThreadImpl.java:176)
> • org.apache.solr.handler.admin.ThreadDumpHandler.handleRequestBody
> (ThreadDumpHandler.java:71)
> • org.apache.solr.handler.RequestHandlerBase.handleRequest
> (RequestHandlerBase.java:199)
> • org.apache.solr.handler.admin.InfoHandler.handle (InfoHandler.java:91)
> • org.apache.solr.handler.admin.InfoHandler.handleRequestBody
> (InfoHandler.java:81)
> • org.apache.solr.handler.RequestHandlerBase.handleRequest
> (RequestHandlerBase.java:199)
> • org.apache.solr.servlet.HttpSolrCall.handleAdmin (HttpSolrCall.java:734)
> • org.apache.solr.servlet.HttpSolrCall.handleAdminRequest
> (HttpSolrCall.java:715)
> • org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:496)
> • org.apache.solr.servlet.SolrDispatchFilter.doFilter
> (SolrDispatchFilter.java:377)
> • org.apache.solr.servlet.SolrDispatchFilter.doFilter
> (SolrDispatchFilter.java:323)
> • org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter
> (ServletHandler.java:1634)
> • org.eclipse.jetty.servlet.ServletHandler.doHandle
> (ServletHandler.java:533)
> • org.eclipse.jetty.server.handler.ScopedHandler.handle
> (ScopedHandler.java:146)
> • org.eclipse.jetty.security.SecurityHandler.handle
> (SecurityHandler.java:548)
> • org.eclipse.jetty.server.handler.HandlerWrapper.handle
> (HandlerWrapper.java:132)
> • org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
> (ScopedHandler.java:257)
> • org.eclipse.jetty.server.session.SessionHandler.doHandle
> (SessionHandler.java:1595)
> • org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
> (ScopedHandler.java:255)
> • org.eclipse.jetty.server.handler.ContextHandler.doHandle
> (ContextHandler.java:1317)
> • org.eclipse.jetty.server.handler.ScopedHandler.nextScope
> (ScopedHandler.java:203)
> • org.eclipse.jetty.servlet.ServletHandler.doScope
> (ServletHandler.java:473)
> • org.eclipse.jetty.server.session.SessionHandler.doScope
> (SessionHandler.java:1564)
> • org.eclipse.jetty.server.handler.ScopedHandler.nextScope
> (ScopedHandler.java:201)
> • org.eclipse.jetty.server.handler.ContextHandler.doScope
> (ContextHandler.java:1219)
> • org.eclipse.jetty.server.handler.ScopedHandler.handle
> (ScopedHandler.java:144)
> • org.eclipse.jetty.server.handler.ContextHandlerCollection.handle
> (ContextHandlerCollection.java:219)
> • org.eclipse.jetty.server.handler.HandlerCollection.handle
> (HandlerCollection.java:126)
> • org.eclipse.jetty.server.handler.HandlerWrapper.handle
> (HandlerWrapper.java:132)
> • org.eclipse.jetty.rewrite.handler.RewriteHandler.handle
> (RewriteHandler.java:335)
> • org.eclipse.jetty.server.handler.HandlerWrapper.handle
> (HandlerWrapper.java:132)
> • org.eclipse.jetty.server.Server.handle (Server.java:531)
> • org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:352)
> • org.eclipse.jetty.server.HttpConnection.onFillable
> (HttpConnection.java:260)
> • org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded
> (AbstractConnection.java:281)
> • org.eclipse.jetty.io.FillInterest.fillable (FillInterest.java:102)
> • org.eclipse.jetty.io.ChannelEndPoint$2.run (ChannelEndPoint.java:118)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask
> (EatWhatYouKill.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:310)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 81.0773ms
> 70.0000ms
> qtp1571967156-114 (114)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 14.7759ms
> 10.0000ms
> qtp1571967156-113 (113)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 1.7541ms
> 0.0000ms
> qtp1571967156-112 (112)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 28.2561ms
> 10.0000ms
> qtp1571967156-109 (109)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 20.5324ms
> 10.0000ms
> qtp1571967156-107 (107)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 13.4180ms
> 0.0000ms
> qtp1571967156-90 (90)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 108.5163ms
> 100.0000ms
> Scheduler-842741472 (89)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@27022346
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> (AbstractQueuedSynchronizer.java:2078)
> • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> (ScheduledThreadPoolExecutor.java:1093)
> • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> (ScheduledThreadPoolExecutor.java:809)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1074)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 5.4794ms
> 0.0000ms
> qtp1571967156-88 (88)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 38.6417ms
> 10.0000ms
> SolrRrdBackendFactory-11-thread-2 (87)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7813101a
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> (AbstractQueuedSynchronizer.java:2078)
> • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> (ScheduledThreadPoolExecutor.java:1093)
> • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> (ScheduledThreadPoolExecutor.java:809)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1074)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 3.3288ms
> 0.0000ms
> searcherExecutor-15-thread-1 (83)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4004f6b2
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2039)
> • java.util.concurrent.LinkedBlockingQueue.take
> (LinkedBlockingQueue.java:442)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1074)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 8.9663ms
> 0.0000ms
> searcherExecutor-14-thread-1 (84)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7397cd2d
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2039)
> • java.util.concurrent.LinkedBlockingQueue.take
> (LinkedBlockingQueue.java:442)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1074)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 10.0677ms
> 0.0000ms
> qtp1571967156-79 (79)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 110.6471ms
> 90.0000ms
> qtp1571967156-78-acceptor-3@6942c593-ServerConnector
> @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (78)
> java.lang.Object@4539d80a
> • sun.nio.ch.ServerSocketChannelImpl.accept
> (ServerSocketChannelImpl.java:234)
> • org.eclipse.jetty.server.ServerConnector.accept
> (ServerConnector.java:369)
> • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
> (AbstractConnector.java:639)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 4.9023ms
> 0.0000ms
> qtp1571967156-77-acceptor-2@26493f4e-ServerConnector
> @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (77)
> java.lang.Object@4539d80a
> • sun.nio.ch.ServerSocketChannelImpl.accept
> (ServerSocketChannelImpl.java:234)
> • org.eclipse.jetty.server.ServerConnector.accept
> (ServerConnector.java:369)
> • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
> (AbstractConnector.java:639)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 5.0835ms
> 0.0000ms
> qtp1571967156-76-acceptor-1@292b8c3e-ServerConnector
> @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (76)
> • sun.nio.ch.ServerSocketChannelImpl.accept0 (Native Method)
> • sun.nio.ch.ServerSocketChannelImpl.accept
> (ServerSocketChannelImpl.java:422)
> • sun.nio.ch.ServerSocketChannelImpl.accept
> (ServerSocketChannelImpl.java:250)
> • org.eclipse.jetty.server.ServerConnector.accept
> (ServerConnector.java:369)
> • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
> (AbstractConnector.java:639)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 5.6738ms
> 0.0000ms
> qtp1571967156-75-acceptor-0@10c4f96f-ServerConnector
> @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (75)
> java.lang.Object@4539d80a
> • sun.nio.ch.ServerSocketChannelImpl.accept
> (ServerSocketChannelImpl.java:234)
> • org.eclipse.jetty.server.ServerConnector.accept
> (ServerConnector.java:369)
> • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
> (AbstractConnector.java:639)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 4.9762ms
> 0.0000ms
> qtp1571967156-74 (74)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> (AbstractQueuedSynchronizer.java:2078)
> • org.eclipse.jetty.util.BlockingArrayQueue.poll
> (BlockingArrayQueue.java:392)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
> (QueuedThreadPool.java:653)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
> (QueuedThreadPool.java:48)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:717)
> • java.lang.Thread.run (Thread.java:748)
> 4.6754ms
> 0.0000ms
> qtp1571967156-73 (73)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> (AbstractQueuedSynchronizer.java:2078)
> • org.eclipse.jetty.util.BlockingArrayQueue.poll
> (BlockingArrayQueue.java:392)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
> (QueuedThreadPool.java:653)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
> (QueuedThreadPool.java:48)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:717)
> • java.lang.Thread.run (Thread.java:748)
> 6.0771ms
> 0.0000ms
> qtp1571967156-72 (72)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 28.1516ms
> 10.0000ms
> qtp1571967156-71 (71)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 31.7190ms
> 10.0000ms
> qtp1571967156-70 (70)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 22.8869ms
> 0.0000ms
> qtp1571967156-69 (69)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> (AbstractQueuedSynchronizer.java:2078)
> • org.eclipse.jetty.util.BlockingArrayQueue.poll
> (BlockingArrayQueue.java:392)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
> (QueuedThreadPool.java:653)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
> (QueuedThreadPool.java:48)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:717)
> • java.lang.Thread.run (Thread.java:748)
> 5.6719ms
> 0.0000ms
> qtp1571967156-68 (68)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 7.0843ms
> 0.0000ms
> qtp1571967156-67 (67)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 25.2077ms
> 10.0000ms
> qtp1571967156-66 (66)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 59.3490ms
> 40.0000ms
> qtp1571967156-65 (65)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 77.9195ms
> 60.0000ms
> qtp1571967156-64 (64)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 44.4205ms
> 30.0000ms
> qtp1571967156-63 (63)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 129.4624ms
> 90.0000ms
> qtp1571967156-62 (62)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@421090bf
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2163)
> •
>
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait
> (ReservedThreadExecutor.java:292)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:357)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 51.2605ms
> 30.0000ms
> DestroyJavaVM (61)
> •
> 2002.8222ms
> 1850.0000ms
> Thread-13 (58)
> java.lang.Object@f2f13c1
> • java.lang.Object.wait (Native Method)
> • java.lang.Object.wait (Object.java:502)
> • org.apache.solr.core.CloserThread.run (CoreContainer.java:1850)
> 0.7894ms
> 0.0000ms
> MetricsHistoryHandler-12-thread-1 (55)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@45d73a6
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> (AbstractQueuedSynchronizer.java:2078)
> • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> (ScheduledThreadPoolExecutor.java:1093)
> • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> (ScheduledThreadPoolExecutor.java:809)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1074)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 348.4029ms
> 310.0000ms
> SolrRrdBackendFactory-11-thread-1 (54)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7813101a
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2039)
> • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> (ScheduledThreadPoolExecutor.java:1088)
> • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> (ScheduledThreadPoolExecutor.java:809)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1074)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 5.7281ms
> 0.0000ms
>
> OverseerAutoScalingTriggerThread-72060990031069185-192.168.1.40:8983_solr-n_0000001018
> (52)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@75fd9961
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2039)
> • org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run
> (OverseerTriggerThread.java:189)
> • java.lang.Thread.run (Thread.java:748)
> 58.1405ms
> 50.0000ms
>
> OverseerCollectionConfigSetProcessor-72060990031069185-192.168.1.40:8983_solr-n_0000001018
> (51)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@55c10b44
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> (AbstractQueuedSynchronizer.java:2078)
> • org.apache.solr.cloud.ZkDistributedQueue.peekElements
> (ZkDistributedQueue.java:463)
> • org.apache.solr.cloud.OverseerTaskQueue.peekTopN
> (OverseerTaskQueue.java:243)
> • org.apache.solr.cloud.OverseerTaskProcessor.run
> (OverseerTaskProcessor.java:233)
> • java.lang.Thread.run (Thread.java:748)
> 220.6397ms
> 170.0000ms
> OverseerStateUpdate-72060990031069185-192.168.1.40:8983_solr-n_0000001018
> (47)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2287ab26
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> (AbstractQueuedSynchronizer.java:2078)
> • org.apache.solr.cloud.ZkDistributedQueue.peekElements
> (ZkDistributedQueue.java:463)
> • org.apache.solr.cloud.Overseer$ClusterStateUpdater.run
> (Overseer.java:207)
> • java.lang.Thread.run (Thread.java:748)
> 189.5649ms
> 140.0000ms
> zkConnectionManagerCallback-13-thread-1 (50)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4da83c93
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2039)
> • java.util.concurrent.LinkedBlockingQueue.take
> (LinkedBlockingQueue.java:442)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1074)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 1.1550ms
> 0.0000ms
> main-EventThread (49)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@540792d6
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2039)
> • java.util.concurrent.LinkedBlockingQueue.take
> (LinkedBlockingQueue.java:442)
> • org.apache.zookeeper.ClientCnxn$EventThread.run (ClientCnxn.java:502)
> 2.6375ms
> 0.0000ms
> main-SendThread(localhost:2181) (48)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • org.apache.zookeeper.ClientCnxnSocketNIO.doTransport
> (ClientCnxnSocketNIO.java:349)
> • org.apache.zookeeper.ClientCnxn$SendThread.run (ClientCnxn.java:1144)
> 52.9289ms
> 10.0000ms
> zkConnectionManagerCallback-6-thread-1 (44)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@9c13839
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2039)
> • java.util.concurrent.LinkedBlockingQueue.take
> (LinkedBlockingQueue.java:442)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1074)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 1.1128ms
> 0.0000ms
> main-EventThread (43)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16806cbc
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2039)
> • java.util.concurrent.LinkedBlockingQueue.take
> (LinkedBlockingQueue.java:442)
> • org.apache.zookeeper.ClientCnxn$EventThread.run (ClientCnxn.java:502)
>
>
> --
>
> --
> Daniel Mantovani
>


--
Regards,
Shalin Shekhar Mangar.
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.6.0 High CPU Usage Deadlock (Generating 70% of futex system call)

Daniel de Oliveira Mantovani
Hello Shalin,

It's not an OS bug,
Linux samba 4.15.0-55-generic #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux

I'm sending the full thread dump log:
2019-11-01 11:05:13
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.231-b11 mixed mode):

"Attach Listener" #259 daemon prio=9 os_prio=0 tid=0x00007fcb78001000
nid=0x2659 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"qtp1571967156-258" #258 prio=5 os_prio=0 tid=0x00007fcabc321000 nid=0x25c7
waiting on condition [0x00007fcb356ca000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005a626cdd8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-257" #257 prio=5 os_prio=0 tid=0x00007fcb24001000 nid=0x25bc
waiting on condition [0x00007fcb35b95000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f00242c0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-256" #256 prio=5 os_prio=0 tid=0x00007fca88003000 nid=0x2594
runnable [0x00007fcb37441000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a34218> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a34200> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0931118> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-254" #254 prio=5 os_prio=0 tid=0x00007fca88005800 nid=0x2570
runnable [0x00007fcb3580f000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a2fdd8> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a2fdc0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0243070> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-252" #252 prio=5 os_prio=0 tid=0x00007fca90008000 nid=0x2441
runnable [0x00007fc59c444000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a30180> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a30168> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0243220> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-250" #250 prio=5 os_prio=0 tid=0x00007fca90001000 nid=0x243d
runnable [0x00007fcb3570b000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a313c8> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a313b0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f06da640> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-249" #249 prio=5 os_prio=0 tid=0x00007fca94001000 nid=0x2435
runnable [0x00007fcb373bf000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a30528> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a30510> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895ec0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-247" #247 prio=5 os_prio=0 tid=0x00007fcb18006000 nid=0x2428
runnable [0x00007fcb372bb000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a31770> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a31758> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f06da6d0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-245" #245 prio=5 os_prio=0 tid=0x00007fcb08004800 nid=0x2417
runnable [0x00007fcb36060000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a31b18> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a31b00> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f06da5b0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-243" #243 prio=5 os_prio=0 tid=0x00007fcae001f000 nid=0x240c
runnable [0x00007fcb35607000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a31ec0> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a31ea8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f06da490> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-241" #241 prio=5 os_prio=0 tid=0x00007fcb14001000 nid=0x23ff
runnable [0x00007fcb3727a000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f08aabe8> (a sun.nio.ch.Util$3)
- locked <0x00000004f08aabd0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0242a88> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-239" #239 prio=5 os_prio=0 tid=0x00007fcb0c001000 nid=0x23f1
runnable [0x00007fcb35648000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f09c4028> (a sun.nio.ch.Util$3)
- locked <0x00000004f09c4010> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895cc8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-237" #237 prio=5 os_prio=0 tid=0x00007fca80001000 nid=0x23de
runnable [0x00007fc59ca8d000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a34098> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a34080> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0930ed8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-235" #235 prio=5 os_prio=0 tid=0x00007fc588026000 nid=0x23d3
runnable [0x00007fcb35e99000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f08aac60> (a sun.nio.ch.Util$3)
- locked <0x00000004f08aac48> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f02429f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-233" #233 prio=5 os_prio=0 tid=0x00007fcacc021000 nid=0x23c6
runnable [0x00007fcb35585000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f08aacc0> (a sun.nio.ch.Util$3)
- locked <0x00000004f08aaca8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0242ba8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-231" #231 prio=5 os_prio=0 tid=0x00007fc554004800 nid=0x23b9
runnable [0x00007fcb35891000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f08aade0> (a sun.nio.ch.Util$3)
- locked <0x00000004f08aadc8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0242ad0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-228" #228 prio=5 os_prio=0 tid=0x00007fcacc015800 nid=0x23a4
runnable [0x00007fcb357ce000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a340f8> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a340e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0930f68> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-226" #226 prio=5 os_prio=0 tid=0x00007fca98005000 nid=0x2394
runnable [0x00007fcb35440000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f09c40e8> (a sun.nio.ch.Util$3)
- locked <0x00000004f09c40d0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895ba8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-224" #224 prio=5 os_prio=0 tid=0x00007fca60016800 nid=0x2388
runnable [0x00007fcb37482000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a34158> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a34140> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0930ff8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-222" #222 prio=5 os_prio=0 tid=0x00007fcb2c003000 nid=0x2378
runnable [0x00007fcb355c6000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f09c4148> (a sun.nio.ch.Util$3)
- locked <0x00000004f09c4130> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895b18> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-220" #220 prio=5 os_prio=0 tid=0x00007fca78003000 nid=0x236a
runnable [0x00007fcb374c3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a341b8> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a341a0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0931088> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-218" #218 prio=5 os_prio=0 tid=0x00007fcb28002000 nid=0x235f
runnable [0x00007fcb360a1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f09c41a8> (a sun.nio.ch.Util$3)
- locked <0x00000004f09c4190> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895a88> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-216" #216 prio=5 os_prio=0 tid=0x00007fcaec002000 nid=0x2353
waiting on condition [0x00007fc59c84c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000405560e38> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-214" #214 prio=5 os_prio=0 tid=0x00007fc58c001800 nid=0x2341
waiting on condition [0x00007fcb37400000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f00242c0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-202" #202 prio=5 os_prio=0 tid=0x00007fca88001000 nid=0x22f3
runnable [0x00007fcb358d2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a31020> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a31008> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895d10> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-200" #200 prio=5 os_prio=0 tid=0x00007fcaa8002800 nid=0x22e7
runnable [0x00007fcb3574c000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a308d0> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a308b8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895e30> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-198" #198 prio=5 os_prio=0 tid=0x00007fca90007800 nid=0x22d7
runnable [0x00007fcb35689000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a30c78> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a30c60> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895da0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"Scheduler-842741472" #89 prio=5 os_prio=0 tid=0x00007fca60007000
nid=0x1c62 waiting on condition [0x00007fc59cace000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f0941af0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"SolrRrdBackendFactory-11-thread-2" #87 prio=5 os_prio=0
tid=0x00007fcac0001000 nid=0x1c60 waiting on condition [0x00007fcb35bd6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f0996978> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"searcherExecutor-15-thread-1" #83 prio=5 os_prio=0 tid=0x00007fcab86b5800
nid=0x18a7 waiting on condition [0x00007fcb34e21000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f45eb9f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"searcherExecutor-14-thread-1" #84 prio=5 os_prio=0 tid=0x00007fcabc9c1000
nid=0x18a6 waiting on condition [0x00007fcb34de0000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f45eb710> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-78-acceptor-3@6942c593-ServerConnector
@362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}" #78 prio=5 os_prio=0
tid=0x00007fca68001000 nid=0x1888 runnable [0x00007fcb35481000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
- locked <0x00000004f08aab70> (a java.lang.Object)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-77-acceptor-2@26493f4e-ServerConnector
@362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}" #77 prio=5 os_prio=0
tid=0x00007fca74001000 nid=0x1886 waiting for monitor entry
[0x00007fcb354c2000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
- waiting to lock <0x00000004f08aab70> (a java.lang.Object)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-76-acceptor-1@292b8c3e-ServerConnector
@362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}" #76 prio=5 os_prio=0
tid=0x00007fca70001000 nid=0x1885 waiting for monitor entry
[0x00007fcb35503000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
- waiting to lock <0x00000004f08aab70> (a java.lang.Object)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-75-acceptor-0@10c4f96f-ServerConnector
@362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}" #75 prio=5 os_prio=0
tid=0x00007fca7c001000 nid=0x1880 waiting for monitor entry
[0x00007fcb35544000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
- waiting to lock <0x00000004f08aab70> (a java.lang.Object)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-66" #66 prio=5 os_prio=0 tid=0x00007fca98001000 nid=0x186f
runnable [0x00007fcb3578d000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f09c4088> (a sun.nio.ch.Util$3)
- locked <0x00000004f09c4070> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895c38> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"DestroyJavaVM" #61 prio=5 os_prio=0 tid=0x00007fcbd800e800 nid=0x175c
waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"Thread-13" #58 prio=5 os_prio=0 tid=0x00007fcbd8f3f800 nid=0x1866 in
Object.wait() [0x00007fcb35b54000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:1850)
- locked <0x00000004f086a0f8> (a java.lang.Object)

   Locked ownable synchronizers:
- None

"MetricsHistoryHandler-12-thread-1" #55 prio=5 os_prio=0
tid=0x00007fcbd8ed3000 nid=0x185e waiting on condition [0x00007fcb35c17000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f08abf80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"SolrRrdBackendFactory-11-thread-1" #54 prio=5 os_prio=0
tid=0x00007fcbd8ed2000 nid=0x185d waiting on condition [0x00007fcb35c58000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f0996978> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"OverseerAutoScalingTriggerThread-72060990031069185-192.168.1.40:8983_solr-n_0000001018"
#52 prio=5 os_prio=0 tid=0x00007fcbd83d8000 nid=0x185b waiting on condition
[0x00007fcb35eda000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f08ac8e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"OverseerCollectionConfigSetProcessor-72060990031069185-192.168.1.40:8983_solr-n_0000001018"
#51 daemon prio=5 os_prio=0 tid=0x00007fcbd83d6800 nid=0x185a runnable
[0x00007fcb35f1b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f09c7b38> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463)
at
org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:243)
at
org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:233)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"OverseerStateUpdate-72060990031069185-192.168.1.40:8983_solr-n_0000001018"
#47 daemon prio=5 os_prio=0 tid=0x00007fcbd83d5800 nid=0x1859 runnable
[0x00007fcb35f5c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f0a2ef20> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463)
at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:207)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"zkConnectionManagerCallback-13-thread-1" #50 prio=5 os_prio=0
tid=0x00007fcad8001000 nid=0x1858 waiting on condition [0x00007fcb35f9d000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f08ac430> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"main-EventThread" #49 daemon prio=5 os_prio=0 tid=0x00007fcbd8941800
nid=0x1857 waiting on condition [0x00007fcb35fde000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f0997bd8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)

   Locked ownable synchronizers:
- None

"main-SendThread(localhost:2181)" #48 daemon prio=5 os_prio=0
tid=0x00007fcbd8948000 nid=0x1856 runnable [0x00007fcb3601f000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0997908> (a sun.nio.ch.Util$3)
- locked <0x00000004f09978f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0242f08> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)

   Locked ownable synchronizers:
- None

"zkConnectionManagerCallback-6-thread-1" #44 prio=5 os_prio=0
tid=0x00007fcaf4001000 nid=0x17a1 waiting on condition [0x00007fcb362e2000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f08a9178> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"main-EventThread" #43 daemon prio=5 os_prio=0 tid=0x00007fcbd8db8800
nid=0x17a0 waiting on condition [0x00007fcb36323000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f08a95c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)

   Locked ownable synchronizers:
- None

"main-SendThread(samba:2181)" #42 daemon prio=5 os_prio=0
tid=0x00007fcbd8db7000 nid=0x179f runnable [0x00007fcb36364000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f08a9898> (a sun.nio.ch.Util$3)
- locked <0x00000004f08a9880> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f03ea3a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)

   Locked ownable synchronizers:
- None

"Connection evictor" #41 daemon prio=5 os_prio=0 tid=0x00007fcbd8daf800
nid=0x179e waiting on condition [0x00007fcb36c79000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"Connection evictor" #40 daemon prio=5 os_prio=0 tid=0x00007fcbd8dae000
nid=0x179d waiting on condition [0x00007fcb36c38000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"Connection evictor" #39 daemon prio=5 os_prio=0 tid=0x00007fcbd8da5000
nid=0x179c waiting on condition [0x00007fcb36cba000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"Scheduler-1456339771" #35 prio=5 os_prio=0 tid=0x00007fcbd8ae3800
nid=0x178e waiting on condition [0x00007fcb36efb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f02dc358> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"ShutdownMonitor" #24 daemon prio=5 os_prio=0 tid=0x00007fcbd86e4800
nid=0x1783 runnable [0x00007fcb37704000]
   java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at
org.eclipse.jetty.server.ShutdownMonitor$ShutdownMonitorRunnable.run(ShutdownMonitor.java:335)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"Service Thread" #21 daemon prio=9 os_prio=0 tid=0x00007fcbd8153800
nid=0x177b runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C1 CompilerThread14" #20 daemon prio=9 os_prio=0 tid=0x00007fcbd8150800
nid=0x177a waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C1 CompilerThread13" #19 daemon prio=9 os_prio=0 tid=0x00007fcbd814e000
nid=0x1779 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C1 CompilerThread12" #18 daemon prio=9 os_prio=0 tid=0x00007fcbd814c800
nid=0x1778 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C1 CompilerThread11" #17 daemon prio=9 os_prio=0 tid=0x00007fcbd814a000
nid=0x1777 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C1 CompilerThread10" #16 daemon prio=9 os_prio=0 tid=0x00007fcbd8148000
nid=0x1776 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread9" #15 daemon prio=9 os_prio=0 tid=0x00007fcbd8146800
nid=0x1775 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread8" #14 daemon prio=9 os_prio=0 tid=0x00007fcbd8144000
nid=0x1774 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread7" #13 daemon prio=9 os_prio=0 tid=0x00007fcbd8141800
nid=0x1773 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread6" #12 daemon prio=9 os_prio=0 tid=0x00007fcbd813f800
nid=0x1772 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread5" #11 daemon prio=9 os_prio=0 tid=0x00007fcbd813d800
nid=0x1771 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread4" #10 daemon prio=9 os_prio=0 tid=0x00007fcbd813b000
nid=0x1770 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread3" #9 daemon prio=9 os_prio=0 tid=0x00007fcbd8131000
nid=0x176f waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbd812f000
nid=0x176e waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbd812c800
nid=0x176d waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbd812a800
nid=0x176c runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbd8128800
nid=0x176b runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
tid=0x00007fcbd8127000 nid=0x176a waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbd80f6800 nid=0x1769 in
Object.wait() [0x00007fcbb807d000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x00000004f03573a0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

   Locked ownable synchronizers:
- None

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbd80f2000
nid=0x1768 in Object.wait() [0x00007fcbdc072000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000004f03574a8> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

   Locked ownable synchronizers:
- None

"VM Thread" os_prio=0 tid=0x00007fcbd80e8800 nid=0x1766 runnable

"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbd8015000
nid=0x175d runnable

"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbd8021800
nid=0x175e runnable

"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbd8023000
nid=0x175f runnable

"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbd8024800
nid=0x1760 runnable

"Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbd806e800
nid=0x1765 runnable

"Gang worker#0 (Parallel CMS Threads)" os_prio=0 tid=0x00007fcbd8067800
nid=0x1761 runnable

"Gang worker#1 (Parallel CMS Threads)" os_prio=0 tid=0x00007fcbd8069000
nid=0x1762 runnable

"Gang worker#2 (Parallel CMS Threads)" os_prio=0 tid=0x00007fcbd806b000
nid=0x1763 runnable

"Gang worker#3 (Parallel CMS Threads)" os_prio=0 tid=0x00007fcbd806c800
nid=0x1764 runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007fcbd8156000 nid=0x177c
waiting on condition

JNI global references: 1002



Thank you

On Fri, Nov 1, 2019 at 8:26 AM Shalin Shekhar Mangar <[hidden email]>
wrote:

> Perhaps this bug in the kernel?
> https://groups.google.com/forum/#!topic/mechanical-sympathy/QbmpZxp6C64
>
> Can you check if you are running an affected kernel?
>
> On Fri, Nov 1, 2019 at 7:09 AM Daniel de Oliveira Mantovani <
> [hidden email]> wrote:
>
> > I have a Solr 7.6.0 Cloud with 4 nodes and while indexing Solr hangs.
> >
> > $ java -version
> > java version "1.8.0_231"
> > Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
> > Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode
> >
> > $ sudo strace -f -c -p 5972
> > % time     seconds  usecs/call     calls    errors syscall
> > ------ ----------- ----------- --------- --------- ----------------
> >  71.02   33.329124         662     50335     13317 futex
> >  21.43   10.055086      335170        30        28 restart_syscall
> >   7.49    3.512846       14760       238           epoll_wait
> >   0.07    0.030526        2544        12           poll
> >   0.00    0.001571           5       322           write
> >   0.00    0.000439           4       117           times
> >   0.00    0.000355           2       183         4 read
> >   0.00    0.000236           3        72           mprotect
> >   0.00    0.000161           1       118           epoll_ctl
> >   0.00    0.000126           0       352           stat
> >   0.00    0.000089          11         8           getdents
> >   0.00    0.000077          10         8           sendto
> >   0.00    0.000069          69         1           statfs
> >   0.00    0.000043          22         2           writev
> >   0.00    0.000029           4         7           openat
> >   0.00    0.000021           4         6           fstat
> >   0.00    0.000018           1        22           lstat
> >   0.00    0.000018           9         2           dup
> >   0.00    0.000010           1         9           close
> >   0.00    0.000010           3         4           fcntl
> >   0.00    0.000005           3         2           access
> >   0.00    0.000002           2         1           mmap
> >   0.00    0.000002           0        26           sched_yield
> >   0.00    0.000001           0         8           recvfrom
> >   0.00    0.000001           1         2           readlink
> > ------ ----------- ----------- --------- --------- ----------------
> > 100.00   46.930865                 51887     13349 total
> >
> > $ sudo strace -f -p 5972
> > strace: Process 5972 attached with 86 threads
> > [pid  7524] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  7523] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  7511] epoll_wait(142,  <unfinished ...>
> > [pid  7471] epoll_wait(160,  <unfinished ...>
> > [pid  7266] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  7267] epoll_wait(151,  <unfinished ...>
> > [pid  7264] futex(0x7fcbd8f3eb98, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  7265] epoll_wait(157,  <unfinished ...>
> > [pid  6311] futex(0x7fcabc8c42f8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6310] futex(0x7fcab843c998, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6281] epoll_wait(154,  <unfinished ...>
> > [pid  6280] futex(0x7fca68001d7c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6278] futex(0x7fca74001d78, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6277] accept(136,  <unfinished ...>
> > [pid  6272] futex(0x7fca7c001d7c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6271] epoll_wait(208,  <unfinished ...>
> > [pid  6270] epoll_wait(205,  <unfinished ...>
> > [pid  6262] epoll_wait(199,  <unfinished ...>
> > [pid  6261] epoll_wait(196,  <unfinished ...>
> > [pid  6258] epoll_wait(193,  <unfinished ...>
> > [pid  6269] epoll_wait(202,  <unfinished ...>
> > [pid  6257] epoll_wait(190,  <unfinished ...>
> > [pid  6256] epoll_wait(187,  <unfinished ...>
> > [pid  6255] epoll_wait(184,  <unfinished ...>
> > [pid  6253] epoll_wait(178,  <unfinished ...>
> > [pid  6250] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  6252] epoll_wait(175,  <unfinished ...>
> > [pid  6251] epoll_wait(172,  <unfinished ...>
> > [pid  6254] epoll_wait(181,  <unfinished ...>
> > [pid  6246] futex(0x7fcbd8f3f378, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6238] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  6237] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  6235] futex(0x7fcbd83d97b8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6234] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  6233] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  6232] futex(0x7fcad8002608, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6231] futex(0x7fcbd89425ac, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6049] futex(0x7fcaf4002388, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6230] epoll_wait(134,  <unfinished ...>
> > [pid  6048] futex(0x7fcbd8db9ac8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6047] epoll_wait(130,  <unfinished ...>
> > [pid  6046] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  6045] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> >
> > JVM args:
> >
> >
> >    -
> >
> -DSTOP.KEY=solrrocks-DSTOP.PORT=7983-Djetty.home=/opt/solr/server-Djetty.port=8983-Dlog4j.configurationFile=file:/opt/solr/server/resources/log4j2.xml-Dsolr.data.home=-Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf-Dsolr.install.dir=/opt/solr-Dsolr.jetty.https.port=8983-Dsolr.log.dir=/opt/solr/example/cloud/node1/solr/../logs-Dsolr.log.muteconsole-Dsolr.solr.home=/opt/solr/example/cloud/node1/solr-Duser.timezone=UTC-DzkClientTimeout=15000-DzkHost=localhost:2181-XX:+CMSParallelRemarkEnabled-XX:+CMSScavengeBeforeRemark-XX:+ParallelRefProcEnabled-XX:+PrintGCApplicationStoppedTime-XX:+PrintGCDateStamps-XX:+PrintGCDetails-XX:+PrintGCTimeStamps-XX:+PrintHeapAtGC-XX:+PrintTenuringDistribution-XX:+UseCMSInitiatingOccupancyOnly-XX:+UseConcMarkSweepGC-XX:+UseGCLogFileRotation-XX:-OmitStackTraceInFastThrow-XX:CMSInitiatingOccupancyFraction=50-XX:CMSMaxAbortablePrecleanTime=6000-XX:ConcGCThreads=4-XX:GCLogFileSize=20M-XX:MaxTenuringThreshold=8-XX:NewRatio=3-XX:NumberOfGCLogFiles=9-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh
> >    8983
> >
> >
> /opt/solr/example/cloud/node1/solr/../logs-XX:ParallelGCThreads=4-XX:PretenureSizeThreshold=64m-XX:SurvivorRatio=4-XX:TargetSurvivorRatio=90-Xloggc:/opt/solr/example/cloud/node1/solr/../logs/solr_gc.log-Xms512m-Xmx15g-Xss256k-verbose:gc
> >
> > Java Thread Dumb:
> > name
> > cpuTime / userTime
> > process reaper (135)
> > java.util.concurrent.SynchronousQueue$TransferStack@70b8631c
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > • java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill
> > (SynchronousQueue.java:460)
> > • java.util.concurrent.SynchronousQueue$TransferStack.transfer
> > (SynchronousQueue.java:362)
> > • java.util.concurrent.SynchronousQueue.poll (SynchronousQueue.java:941)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1073)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 17.2219ms
> > 0.0000ms
> > qtp1571967156-129 (129)
> > 36.0159ms
> > 0.0000ms
> > process reaper (117)
> > java.util.concurrent.SynchronousQueue$TransferStack@70b8631c
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > • java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill
> > (SynchronousQueue.java:460)
> > • java.util.concurrent.SynchronousQueue$TransferStack.transfer
> > (SynchronousQueue.java:362)
> > • java.util.concurrent.SynchronousQueue.poll (SynchronousQueue.java:941)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1073)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 19.2751ms
> > 0.0000ms
> > qtp1571967156-116 (116)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 1.5766ms
> > 0.0000ms
> > qtp1571967156-115 (115)
> > • sun.management.ThreadImpl.getThreadInfo1 (Native Method)
> > • sun.management.ThreadImpl.getThreadInfo (ThreadImpl.java:176)
> > • org.apache.solr.handler.admin.ThreadDumpHandler.handleRequestBody
> > (ThreadDumpHandler.java:71)
> > • org.apache.solr.handler.RequestHandlerBase.handleRequest
> > (RequestHandlerBase.java:199)
> > • org.apache.solr.handler.admin.InfoHandler.handle (InfoHandler.java:91)
> > • org.apache.solr.handler.admin.InfoHandler.handleRequestBody
> > (InfoHandler.java:81)
> > • org.apache.solr.handler.RequestHandlerBase.handleRequest
> > (RequestHandlerBase.java:199)
> > • org.apache.solr.servlet.HttpSolrCall.handleAdmin
> (HttpSolrCall.java:734)
> > • org.apache.solr.servlet.HttpSolrCall.handleAdminRequest
> > (HttpSolrCall.java:715)
> > • org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:496)
> > • org.apache.solr.servlet.SolrDispatchFilter.doFilter
> > (SolrDispatchFilter.java:377)
> > • org.apache.solr.servlet.SolrDispatchFilter.doFilter
> > (SolrDispatchFilter.java:323)
> > • org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter
> > (ServletHandler.java:1634)
> > • org.eclipse.jetty.servlet.ServletHandler.doHandle
> > (ServletHandler.java:533)
> > • org.eclipse.jetty.server.handler.ScopedHandler.handle
> > (ScopedHandler.java:146)
> > • org.eclipse.jetty.security.SecurityHandler.handle
> > (SecurityHandler.java:548)
> > • org.eclipse.jetty.server.handler.HandlerWrapper.handle
> > (HandlerWrapper.java:132)
> > • org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
> > (ScopedHandler.java:257)
> > • org.eclipse.jetty.server.session.SessionHandler.doHandle
> > (SessionHandler.java:1595)
> > • org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
> > (ScopedHandler.java:255)
> > • org.eclipse.jetty.server.handler.ContextHandler.doHandle
> > (ContextHandler.java:1317)
> > • org.eclipse.jetty.server.handler.ScopedHandler.nextScope
> > (ScopedHandler.java:203)
> > • org.eclipse.jetty.servlet.ServletHandler.doScope
> > (ServletHandler.java:473)
> > • org.eclipse.jetty.server.session.SessionHandler.doScope
> > (SessionHandler.java:1564)
> > • org.eclipse.jetty.server.handler.ScopedHandler.nextScope
> > (ScopedHandler.java:201)
> > • org.eclipse.jetty.server.handler.ContextHandler.doScope
> > (ContextHandler.java:1219)
> > • org.eclipse.jetty.server.handler.ScopedHandler.handle
> > (ScopedHandler.java:144)
> > • org.eclipse.jetty.server.handler.ContextHandlerCollection.handle
> > (ContextHandlerCollection.java:219)
> > • org.eclipse.jetty.server.handler.HandlerCollection.handle
> > (HandlerCollection.java:126)
> > • org.eclipse.jetty.server.handler.HandlerWrapper.handle
> > (HandlerWrapper.java:132)
> > • org.eclipse.jetty.rewrite.handler.RewriteHandler.handle
> > (RewriteHandler.java:335)
> > • org.eclipse.jetty.server.handler.HandlerWrapper.handle
> > (HandlerWrapper.java:132)
> > • org.eclipse.jetty.server.Server.handle (Server.java:531)
> > • org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:352)
> > • org.eclipse.jetty.server.HttpConnection.onFillable
> > (HttpConnection.java:260)
> > • org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded
> > (AbstractConnection.java:281)
> > • org.eclipse.jetty.io.FillInterest.fillable (FillInterest.java:102)
> > • org.eclipse.jetty.io.ChannelEndPoint$2.run (ChannelEndPoint.java:118)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask
> > (EatWhatYouKill.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:310)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 81.0773ms
> > 70.0000ms
> > qtp1571967156-114 (114)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 14.7759ms
> > 10.0000ms
> > qtp1571967156-113 (113)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 1.7541ms
> > 0.0000ms
> > qtp1571967156-112 (112)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 28.2561ms
> > 10.0000ms
> > qtp1571967156-109 (109)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 20.5324ms
> > 10.0000ms
> > qtp1571967156-107 (107)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 13.4180ms
> > 0.0000ms
> > qtp1571967156-90 (90)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 108.5163ms
> > 100.0000ms
> > Scheduler-842741472 (89)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@27022346
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> > (AbstractQueuedSynchronizer.java:2078)
> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> > (ScheduledThreadPoolExecutor.java:1093)
> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> > (ScheduledThreadPoolExecutor.java:809)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1074)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 5.4794ms
> > 0.0000ms
> > qtp1571967156-88 (88)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 38.6417ms
> > 10.0000ms
> > SolrRrdBackendFactory-11-thread-2 (87)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7813101a
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> > (AbstractQueuedSynchronizer.java:2078)
> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> > (ScheduledThreadPoolExecutor.java:1093)
> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> > (ScheduledThreadPoolExecutor.java:809)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1074)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 3.3288ms
> > 0.0000ms
> > searcherExecutor-15-thread-1 (83)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4004f6b2
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2039)
> > • java.util.concurrent.LinkedBlockingQueue.take
> > (LinkedBlockingQueue.java:442)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1074)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 8.9663ms
> > 0.0000ms
> > searcherExecutor-14-thread-1 (84)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7397cd2d
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2039)
> > • java.util.concurrent.LinkedBlockingQueue.take
> > (LinkedBlockingQueue.java:442)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1074)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 10.0677ms
> > 0.0000ms
> > qtp1571967156-79 (79)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 110.6471ms
> > 90.0000ms
> > qtp1571967156-78-acceptor-3@6942c593-ServerConnector
> > @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (78)
> > java.lang.Object@4539d80a
> > • sun.nio.ch.ServerSocketChannelImpl.accept
> > (ServerSocketChannelImpl.java:234)
> > • org.eclipse.jetty.server.ServerConnector.accept
> > (ServerConnector.java:369)
> > • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
> > (AbstractConnector.java:639)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 4.9023ms
> > 0.0000ms
> > qtp1571967156-77-acceptor-2@26493f4e-ServerConnector
> > @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (77)
> > java.lang.Object@4539d80a
> > • sun.nio.ch.ServerSocketChannelImpl.accept
> > (ServerSocketChannelImpl.java:234)
> > • org.eclipse.jetty.server.ServerConnector.accept
> > (ServerConnector.java:369)
> > • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
> > (AbstractConnector.java:639)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 5.0835ms
> > 0.0000ms
> > qtp1571967156-76-acceptor-1@292b8c3e-ServerConnector
> > @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (76)
> > • sun.nio.ch.ServerSocketChannelImpl.accept0 (Native Method)
> > • sun.nio.ch.ServerSocketChannelImpl.accept
> > (ServerSocketChannelImpl.java:422)
> > • sun.nio.ch.ServerSocketChannelImpl.accept
> > (ServerSocketChannelImpl.java:250)
> > • org.eclipse.jetty.server.ServerConnector.accept
> > (ServerConnector.java:369)
> > • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
> > (AbstractConnector.java:639)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 5.6738ms
> > 0.0000ms
> > qtp1571967156-75-acceptor-0@10c4f96f-ServerConnector
> > @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (75)
> > java.lang.Object@4539d80a
> > • sun.nio.ch.ServerSocketChannelImpl.accept
> > (ServerSocketChannelImpl.java:234)
> > • org.eclipse.jetty.server.ServerConnector.accept
> > (ServerConnector.java:369)
> > • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
> > (AbstractConnector.java:639)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 4.9762ms
> > 0.0000ms
> > qtp1571967156-74 (74)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> > (AbstractQueuedSynchronizer.java:2078)
> > • org.eclipse.jetty.util.BlockingArrayQueue.poll
> > (BlockingArrayQueue.java:392)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
> > (QueuedThreadPool.java:653)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
> > (QueuedThreadPool.java:48)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:717)
> > • java.lang.Thread.run (Thread.java:748)
> > 4.6754ms
> > 0.0000ms
> > qtp1571967156-73 (73)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> > (AbstractQueuedSynchronizer.java:2078)
> > • org.eclipse.jetty.util.BlockingArrayQueue.poll
> > (BlockingArrayQueue.java:392)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
> > (QueuedThreadPool.java:653)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
> > (QueuedThreadPool.java:48)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:717)
> > • java.lang.Thread.run (Thread.java:748)
> > 6.0771ms
> > 0.0000ms
> > qtp1571967156-72 (72)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 28.1516ms
> > 10.0000ms
> > qtp1571967156-71 (71)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 31.7190ms
> > 10.0000ms
> > qtp1571967156-70 (70)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 22.8869ms
> > 0.0000ms
> > qtp1571967156-69 (69)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> > (AbstractQueuedSynchronizer.java:2078)
> > • org.eclipse.jetty.util.BlockingArrayQueue.poll
> > (BlockingArrayQueue.java:392)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
> > (QueuedThreadPool.java:653)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
> > (QueuedThreadPool.java:48)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:717)
> > • java.lang.Thread.run (Thread.java:748)
> > 5.6719ms
> > 0.0000ms
> > qtp1571967156-68 (68)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 7.0843ms
> > 0.0000ms
> > qtp1571967156-67 (67)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 25.2077ms
> > 10.0000ms
> > qtp1571967156-66 (66)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 59.3490ms
> > 40.0000ms
> > qtp1571967156-65 (65)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 77.9195ms
> > 60.0000ms
> > qtp1571967156-64 (64)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 44.4205ms
> > 30.0000ms
> > qtp1571967156-63 (63)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 129.4624ms
> > 90.0000ms
> > qtp1571967156-62 (62)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@421090bf
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2163)
> > •
> >
> >
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait
> > (ReservedThreadExecutor.java:292)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:357)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 51.2605ms
> > 30.0000ms
> > DestroyJavaVM (61)
> > •
> > 2002.8222ms
> > 1850.0000ms
> > Thread-13 (58)
> > java.lang.Object@f2f13c1
> > • java.lang.Object.wait (Native Method)
> > • java.lang.Object.wait (Object.java:502)
> > • org.apache.solr.core.CloserThread.run (CoreContainer.java:1850)
> > 0.7894ms
> > 0.0000ms
> > MetricsHistoryHandler-12-thread-1 (55)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@45d73a6
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> > (AbstractQueuedSynchronizer.java:2078)
> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> > (ScheduledThreadPoolExecutor.java:1093)
> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> > (ScheduledThreadPoolExecutor.java:809)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1074)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 348.4029ms
> > 310.0000ms
> > SolrRrdBackendFactory-11-thread-1 (54)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7813101a
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2039)
> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> > (ScheduledThreadPoolExecutor.java:1088)
> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> > (ScheduledThreadPoolExecutor.java:809)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1074)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 5.7281ms
> > 0.0000ms
> >
> >
> OverseerAutoScalingTriggerThread-72060990031069185-192.168.1.40:8983_solr-n_0000001018
> > (52)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@75fd9961
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2039)
> > • org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run
> > (OverseerTriggerThread.java:189)
> > • java.lang.Thread.run (Thread.java:748)
> > 58.1405ms
> > 50.0000ms
> >
> >
> OverseerCollectionConfigSetProcessor-72060990031069185-192.168.1.40:8983_solr-n_0000001018
> > (51)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@55c10b44
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> > (AbstractQueuedSynchronizer.java:2078)
> > • org.apache.solr.cloud.ZkDistributedQueue.peekElements
> > (ZkDistributedQueue.java:463)
> > • org.apache.solr.cloud.OverseerTaskQueue.peekTopN
> > (OverseerTaskQueue.java:243)
> > • org.apache.solr.cloud.OverseerTaskProcessor.run
> > (OverseerTaskProcessor.java:233)
> > • java.lang.Thread.run (Thread.java:748)
> > 220.6397ms
> > 170.0000ms
> > OverseerStateUpdate-72060990031069185-192.168.1.40:8983_solr-n_0000001018
> > (47)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2287ab26
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> > (AbstractQueuedSynchronizer.java:2078)
> > • org.apache.solr.cloud.ZkDistributedQueue.peekElements
> > (ZkDistributedQueue.java:463)
> > • org.apache.solr.cloud.Overseer$ClusterStateUpdater.run
> > (Overseer.java:207)
> > • java.lang.Thread.run (Thread.java:748)
> > 189.5649ms
> > 140.0000ms
> > zkConnectionManagerCallback-13-thread-1 (50)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4da83c93
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2039)
> > • java.util.concurrent.LinkedBlockingQueue.take
> > (LinkedBlockingQueue.java:442)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1074)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 1.1550ms
> > 0.0000ms
> > main-EventThread (49)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@540792d6
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2039)
> > • java.util.concurrent.LinkedBlockingQueue.take
> > (LinkedBlockingQueue.java:442)
> > • org.apache.zookeeper.ClientCnxn$EventThread.run (ClientCnxn.java:502)
> > 2.6375ms
> > 0.0000ms
> > main-SendThread(localhost:2181) (48)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • org.apache.zookeeper.ClientCnxnSocketNIO.doTransport
> > (ClientCnxnSocketNIO.java:349)
> > • org.apache.zookeeper.ClientCnxn$SendThread.run (ClientCnxn.java:1144)
> > 52.9289ms
> > 10.0000ms
> > zkConnectionManagerCallback-6-thread-1 (44)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@9c13839
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2039)
> > • java.util.concurrent.LinkedBlockingQueue.take
> > (LinkedBlockingQueue.java:442)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1074)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 1.1128ms
> > 0.0000ms
> > main-EventThread (43)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16806cbc
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2039)
> > • java.util.concurrent.LinkedBlockingQueue.take
> > (LinkedBlockingQueue.java:442)
> > • org.apache.zookeeper.ClientCnxn$EventThread.run (ClientCnxn.java:502)
> >
> >
> > --
> >
> > --
> > Daniel Mantovani
> >
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>


--

--
Daniel Mantovani
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.6.0 High CPU Usage Deadlock (Generating 70% of futex system call)

Daniel de Oliveira Mantovani
I set up Solr Cloud 7.7.2 and follow the tutorial
https://lucene.apache.org/solr/guide/8_2/getting-started-with-solrcloud.html#getting-started-with-solrcloud

I still see exactly the same behavior, even with 0 records and out of the
box. Why would it generates 40.000 Futex syscall in 30 seconds even if
there's no record on it ?



On Fri, Nov 1, 2019 at 9:15 AM Daniel de Oliveira Mantovani <
[hidden email]> wrote:

> Hello Shalin,
>
> It's not an OS bug,
> Linux samba 4.15.0-55-generic #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019
> x86_64 x86_64 x86_64 GNU/Linux
>
> I'm sending the full thread dump log:
> 2019-11-01 11:05:13
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.231-b11 mixed mode):
>
> "Attach Listener" #259 daemon prio=9 os_prio=0 tid=0x00007fcb78001000
> nid=0x2659 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-258" #258 prio=5 os_prio=0 tid=0x00007fcabc321000
> nid=0x25c7 waiting on condition [0x00007fcb356ca000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005a626cdd8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-257" #257 prio=5 os_prio=0 tid=0x00007fcb24001000
> nid=0x25bc waiting on condition [0x00007fcb35b95000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000004f00242c0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-256" #256 prio=5 os_prio=0 tid=0x00007fca88003000
> nid=0x2594 runnable [0x00007fcb37441000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f0a34218> (a sun.nio.ch.Util$3)
> - locked <0x00000004f0a34200> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0931118> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-254" #254 prio=5 os_prio=0 tid=0x00007fca88005800
> nid=0x2570 runnable [0x00007fcb3580f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f0a2fdd8> (a sun.nio.ch.Util$3)
> - locked <0x00000004f0a2fdc0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0243070> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-252" #252 prio=5 os_prio=0 tid=0x00007fca90008000
> nid=0x2441 runnable [0x00007fc59c444000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f0a30180> (a sun.nio.ch.Util$3)
> - locked <0x00000004f0a30168> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0243220> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-250" #250 prio=5 os_prio=0 tid=0x00007fca90001000
> nid=0x243d runnable [0x00007fcb3570b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f0a313c8> (a sun.nio.ch.Util$3)
> - locked <0x00000004f0a313b0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f06da640> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-249" #249 prio=5 os_prio=0 tid=0x00007fca94001000
> nid=0x2435 runnable [0x00007fcb373bf000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f0a30528> (a sun.nio.ch.Util$3)
> - locked <0x00000004f0a30510> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0895ec0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-247" #247 prio=5 os_prio=0 tid=0x00007fcb18006000
> nid=0x2428 runnable [0x00007fcb372bb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f0a31770> (a sun.nio.ch.Util$3)
> - locked <0x00000004f0a31758> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f06da6d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-245" #245 prio=5 os_prio=0 tid=0x00007fcb08004800
> nid=0x2417 runnable [0x00007fcb36060000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f0a31b18> (a sun.nio.ch.Util$3)
> - locked <0x00000004f0a31b00> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f06da5b0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-243" #243 prio=5 os_prio=0 tid=0x00007fcae001f000
> nid=0x240c runnable [0x00007fcb35607000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f0a31ec0> (a sun.nio.ch.Util$3)
> - locked <0x00000004f0a31ea8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f06da490> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-241" #241 prio=5 os_prio=0 tid=0x00007fcb14001000
> nid=0x23ff runnable [0x00007fcb3727a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f08aabe8> (a sun.nio.ch.Util$3)
> - locked <0x00000004f08aabd0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0242a88> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-239" #239 prio=5 os_prio=0 tid=0x00007fcb0c001000
> nid=0x23f1 runnable [0x00007fcb35648000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f09c4028> (a sun.nio.ch.Util$3)
> - locked <0x00000004f09c4010> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0895cc8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-237" #237 prio=5 os_prio=0 tid=0x00007fca80001000
> nid=0x23de runnable [0x00007fc59ca8d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f0a34098> (a sun.nio.ch.Util$3)
> - locked <0x00000004f0a34080> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0930ed8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-235" #235 prio=5 os_prio=0 tid=0x00007fc588026000
> nid=0x23d3 runnable [0x00007fcb35e99000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f08aac60> (a sun.nio.ch.Util$3)
> - locked <0x00000004f08aac48> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f02429f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-233" #233 prio=5 os_prio=0 tid=0x00007fcacc021000
> nid=0x23c6 runnable [0x00007fcb35585000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f08aacc0> (a sun.nio.ch.Util$3)
> - locked <0x00000004f08aaca8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0242ba8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-231" #231 prio=5 os_prio=0 tid=0x00007fc554004800
> nid=0x23b9 runnable [0x00007fcb35891000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f08aade0> (a sun.nio.ch.Util$3)
> - locked <0x00000004f08aadc8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0242ad0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-228" #228 prio=5 os_prio=0 tid=0x00007fcacc015800
> nid=0x23a4 runnable [0x00007fcb357ce000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f0a340f8> (a sun.nio.ch.Util$3)
> - locked <0x00000004f0a340e0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0930f68> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-226" #226 prio=5 os_prio=0 tid=0x00007fca98005000
> nid=0x2394 runnable [0x00007fcb35440000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f09c40e8> (a sun.nio.ch.Util$3)
> - locked <0x00000004f09c40d0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0895ba8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-224" #224 prio=5 os_prio=0 tid=0x00007fca60016800
> nid=0x2388 runnable [0x00007fcb37482000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f0a34158> (a sun.nio.ch.Util$3)
> - locked <0x00000004f0a34140> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0930ff8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-222" #222 prio=5 os_prio=0 tid=0x00007fcb2c003000
> nid=0x2378 runnable [0x00007fcb355c6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f09c4148> (a sun.nio.ch.Util$3)
> - locked <0x00000004f09c4130> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0895b18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-220" #220 prio=5 os_prio=0 tid=0x00007fca78003000
> nid=0x236a runnable [0x00007fcb374c3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f0a341b8> (a sun.nio.ch.Util$3)
> - locked <0x00000004f0a341a0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0931088> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-218" #218 prio=5 os_prio=0 tid=0x00007fcb28002000
> nid=0x235f runnable [0x00007fcb360a1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f09c41a8> (a sun.nio.ch.Util$3)
> - locked <0x00000004f09c4190> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0895a88> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-216" #216 prio=5 os_prio=0 tid=0x00007fcaec002000
> nid=0x2353 waiting on condition [0x00007fc59c84c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000405560e38> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-214" #214 prio=5 os_prio=0 tid=0x00007fc58c001800
> nid=0x2341 waiting on condition [0x00007fcb37400000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000004f00242c0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-202" #202 prio=5 os_prio=0 tid=0x00007fca88001000
> nid=0x22f3 runnable [0x00007fcb358d2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f0a31020> (a sun.nio.ch.Util$3)
> - locked <0x00000004f0a31008> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0895d10> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-200" #200 prio=5 os_prio=0 tid=0x00007fcaa8002800
> nid=0x22e7 runnable [0x00007fcb3574c000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f0a308d0> (a sun.nio.ch.Util$3)
> - locked <0x00000004f0a308b8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0895e30> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-198" #198 prio=5 os_prio=0 tid=0x00007fca90007800
> nid=0x22d7 runnable [0x00007fcb35689000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f0a30c78> (a sun.nio.ch.Util$3)
> - locked <0x00000004f0a30c60> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0895da0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "Scheduler-842741472" #89 prio=5 os_prio=0 tid=0x00007fca60007000
> nid=0x1c62 waiting on condition [0x00007fc59cace000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000004f0941af0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "SolrRrdBackendFactory-11-thread-2" #87 prio=5 os_prio=0
> tid=0x00007fcac0001000 nid=0x1c60 waiting on condition [0x00007fcb35bd6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000004f0996978> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "searcherExecutor-15-thread-1" #83 prio=5 os_prio=0 tid=0x00007fcab86b5800
> nid=0x18a7 waiting on condition [0x00007fcb34e21000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000004f45eb9f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "searcherExecutor-14-thread-1" #84 prio=5 os_prio=0 tid=0x00007fcabc9c1000
> nid=0x18a6 waiting on condition [0x00007fcb34de0000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000004f45eb710> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-78-acceptor-3@6942c593-ServerConnector
> @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}" #78 prio=5 os_prio=0
> tid=0x00007fca68001000 nid=0x1888 runnable [0x00007fcb35481000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> - locked <0x00000004f08aab70> (a java.lang.Object)
> at
> org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
> at
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-77-acceptor-2@26493f4e-ServerConnector
> @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}" #77 prio=5 os_prio=0
> tid=0x00007fca74001000 nid=0x1886 waiting for monitor entry
> [0x00007fcb354c2000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
> - waiting to lock <0x00000004f08aab70> (a java.lang.Object)
> at
> org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
> at
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-76-acceptor-1@292b8c3e-ServerConnector
> @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}" #76 prio=5 os_prio=0
> tid=0x00007fca70001000 nid=0x1885 waiting for monitor entry
> [0x00007fcb35503000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
> - waiting to lock <0x00000004f08aab70> (a java.lang.Object)
> at
> org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
> at
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-75-acceptor-0@10c4f96f-ServerConnector
> @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}" #75 prio=5 os_prio=0
> tid=0x00007fca7c001000 nid=0x1880 waiting for monitor entry
> [0x00007fcb35544000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
> - waiting to lock <0x00000004f08aab70> (a java.lang.Object)
> at
> org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
> at
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "qtp1571967156-66" #66 prio=5 os_prio=0 tid=0x00007fca98001000 nid=0x186f
> runnable [0x00007fcb3578d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f09c4088> (a sun.nio.ch.Util$3)
> - locked <0x00000004f09c4070> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0895c38> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "DestroyJavaVM" #61 prio=5 os_prio=0 tid=0x00007fcbd800e800 nid=0x175c
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "Thread-13" #58 prio=5 os_prio=0 tid=0x00007fcbd8f3f800 nid=0x1866 in
> Object.wait() [0x00007fcb35b54000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.solr.core.CloserThread.run(CoreContainer.java:1850)
> - locked <0x00000004f086a0f8> (a java.lang.Object)
>
>    Locked ownable synchronizers:
> - None
>
> "MetricsHistoryHandler-12-thread-1" #55 prio=5 os_prio=0
> tid=0x00007fcbd8ed3000 nid=0x185e waiting on condition [0x00007fcb35c17000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000004f08abf80> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "SolrRrdBackendFactory-11-thread-1" #54 prio=5 os_prio=0
> tid=0x00007fcbd8ed2000 nid=0x185d waiting on condition [0x00007fcb35c58000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000004f0996978> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "OverseerAutoScalingTriggerThread-72060990031069185-192.168.1.40:8983_solr-n_0000001018"
> #52 prio=5 os_prio=0 tid=0x00007fcbd83d8000 nid=0x185b waiting on condition
> [0x00007fcb35eda000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000004f08ac8e0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "OverseerCollectionConfigSetProcessor-72060990031069185-192.168.1.40:8983_solr-n_0000001018"
> #51 daemon prio=5 os_prio=0 tid=0x00007fcbd83d6800 nid=0x185a runnable
> [0x00007fcb35f1b000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000004f09c7b38> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463)
> at
> org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:243)
> at
> org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:233)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "OverseerStateUpdate-72060990031069185-192.168.1.40:8983_solr-n_0000001018"
> #47 daemon prio=5 os_prio=0 tid=0x00007fcbd83d5800 nid=0x1859 runnable
> [0x00007fcb35f5c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000004f0a2ef20> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463)
> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:207)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "zkConnectionManagerCallback-13-thread-1" #50 prio=5 os_prio=0
> tid=0x00007fcad8001000 nid=0x1858 waiting on condition [0x00007fcb35f9d000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000004f08ac430> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "main-EventThread" #49 daemon prio=5 os_prio=0 tid=0x00007fcbd8941800
> nid=0x1857 waiting on condition [0x00007fcb35fde000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000004f0997bd8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>
>    Locked ownable synchronizers:
> - None
>
> "main-SendThread(localhost:2181)" #48 daemon prio=5 os_prio=0
> tid=0x00007fcbd8948000 nid=0x1856 runnable [0x00007fcb3601f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f0997908> (a sun.nio.ch.Util$3)
> - locked <0x00000004f09978f0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f0242f08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
>
>    Locked ownable synchronizers:
> - None
>
> "zkConnectionManagerCallback-6-thread-1" #44 prio=5 os_prio=0
> tid=0x00007fcaf4001000 nid=0x17a1 waiting on condition [0x00007fcb362e2000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000004f08a9178> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "main-EventThread" #43 daemon prio=5 os_prio=0 tid=0x00007fcbd8db8800
> nid=0x17a0 waiting on condition [0x00007fcb36323000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000004f08a95c8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>
>    Locked ownable synchronizers:
> - None
>
> "main-SendThread(samba:2181)" #42 daemon prio=5 os_prio=0
> tid=0x00007fcbd8db7000 nid=0x179f runnable [0x00007fcb36364000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000004f08a9898> (a sun.nio.ch.Util$3)
> - locked <0x00000004f08a9880> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000004f03ea3a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
>
>    Locked ownable synchronizers:
> - None
>
> "Connection evictor" #41 daemon prio=5 os_prio=0 tid=0x00007fcbd8daf800
> nid=0x179e waiting on condition [0x00007fcb36c79000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "Connection evictor" #40 daemon prio=5 os_prio=0 tid=0x00007fcbd8dae000
> nid=0x179d waiting on condition [0x00007fcb36c38000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "Connection evictor" #39 daemon prio=5 os_prio=0 tid=0x00007fcbd8da5000
> nid=0x179c waiting on condition [0x00007fcb36cba000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "Scheduler-1456339771" #35 prio=5 os_prio=0 tid=0x00007fcbd8ae3800
> nid=0x178e waiting on condition [0x00007fcb36efb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000004f02dc358> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "ShutdownMonitor" #24 daemon prio=5 os_prio=0 tid=0x00007fcbd86e4800
> nid=0x1783 runnable [0x00007fcb37704000]
>    java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at
> org.eclipse.jetty.server.ShutdownMonitor$ShutdownMonitorRunnable.run(ShutdownMonitor.java:335)
> at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
> - None
>
> "Service Thread" #21 daemon prio=9 os_prio=0 tid=0x00007fcbd8153800
> nid=0x177b runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "C1 CompilerThread14" #20 daemon prio=9 os_prio=0 tid=0x00007fcbd8150800
> nid=0x177a waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "C1 CompilerThread13" #19 daemon prio=9 os_prio=0 tid=0x00007fcbd814e000
> nid=0x1779 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "C1 CompilerThread12" #18 daemon prio=9 os_prio=0 tid=0x00007fcbd814c800
> nid=0x1778 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "C1 CompilerThread11" #17 daemon prio=9 os_prio=0 tid=0x00007fcbd814a000
> nid=0x1777 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "C1 CompilerThread10" #16 daemon prio=9 os_prio=0 tid=0x00007fcbd8148000
> nid=0x1776 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "C2 CompilerThread9" #15 daemon prio=9 os_prio=0 tid=0x00007fcbd8146800
> nid=0x1775 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "C2 CompilerThread8" #14 daemon prio=9 os_prio=0 tid=0x00007fcbd8144000
> nid=0x1774 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "C2 CompilerThread7" #13 daemon prio=9 os_prio=0 tid=0x00007fcbd8141800
> nid=0x1773 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "C2 CompilerThread6" #12 daemon prio=9 os_prio=0 tid=0x00007fcbd813f800
> nid=0x1772 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "C2 CompilerThread5" #11 daemon prio=9 os_prio=0 tid=0x00007fcbd813d800
> nid=0x1771 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "C2 CompilerThread4" #10 daemon prio=9 os_prio=0 tid=0x00007fcbd813b000
> nid=0x1770 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "C2 CompilerThread3" #9 daemon prio=9 os_prio=0 tid=0x00007fcbd8131000
> nid=0x176f waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "C2 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbd812f000
> nid=0x176e waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbd812c800
> nid=0x176d waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbd812a800
> nid=0x176c runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbd8128800
> nid=0x176b runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007fcbd8127000 nid=0x176a waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
> - None
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbd80f6800 nid=0x1769
> in Object.wait() [0x00007fcbb807d000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
> - locked <0x00000004f03573a0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
>
>    Locked ownable synchronizers:
> - None
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbd80f2000
> nid=0x1768 in Object.wait() [0x00007fcbdc072000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> - locked <0x00000004f03574a8> (a java.lang.ref.Reference$Lock)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
>
>    Locked ownable synchronizers:
> - None
>
> "VM Thread" os_prio=0 tid=0x00007fcbd80e8800 nid=0x1766 runnable
>
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbd8015000
> nid=0x175d runnable
>
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbd8021800
> nid=0x175e runnable
>
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbd8023000
> nid=0x175f runnable
>
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbd8024800
> nid=0x1760 runnable
>
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbd806e800
> nid=0x1765 runnable
>
> "Gang worker#0 (Parallel CMS Threads)" os_prio=0 tid=0x00007fcbd8067800
> nid=0x1761 runnable
>
> "Gang worker#1 (Parallel CMS Threads)" os_prio=0 tid=0x00007fcbd8069000
> nid=0x1762 runnable
>
> "Gang worker#2 (Parallel CMS Threads)" os_prio=0 tid=0x00007fcbd806b000
> nid=0x1763 runnable
>
> "Gang worker#3 (Parallel CMS Threads)" os_prio=0 tid=0x00007fcbd806c800
> nid=0x1764 runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbd8156000 nid=0x177c
> waiting on condition
>
> JNI global references: 1002
>
>
>
> Thank you
>
> On Fri, Nov 1, 2019 at 8:26 AM Shalin Shekhar Mangar <
> [hidden email]> wrote:
>
>> Perhaps this bug in the kernel?
>> https://groups.google.com/forum/#!topic/mechanical-sympathy/QbmpZxp6C64
>>
>> Can you check if you are running an affected kernel?
>>
>> On Fri, Nov 1, 2019 at 7:09 AM Daniel de Oliveira Mantovani <
>> [hidden email]> wrote:
>>
>> > I have a Solr 7.6.0 Cloud with 4 nodes and while indexing Solr hangs.
>> >
>> > $ java -version
>> > java version "1.8.0_231"
>> > Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
>> > Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode
>> >
>> > $ sudo strace -f -c -p 5972
>> > % time     seconds  usecs/call     calls    errors syscall
>> > ------ ----------- ----------- --------- --------- ----------------
>> >  71.02   33.329124         662     50335     13317 futex
>> >  21.43   10.055086      335170        30        28 restart_syscall
>> >   7.49    3.512846       14760       238           epoll_wait
>> >   0.07    0.030526        2544        12           poll
>> >   0.00    0.001571           5       322           write
>> >   0.00    0.000439           4       117           times
>> >   0.00    0.000355           2       183         4 read
>> >   0.00    0.000236           3        72           mprotect
>> >   0.00    0.000161           1       118           epoll_ctl
>> >   0.00    0.000126           0       352           stat
>> >   0.00    0.000089          11         8           getdents
>> >   0.00    0.000077          10         8           sendto
>> >   0.00    0.000069          69         1           statfs
>> >   0.00    0.000043          22         2           writev
>> >   0.00    0.000029           4         7           openat
>> >   0.00    0.000021           4         6           fstat
>> >   0.00    0.000018           1        22           lstat
>> >   0.00    0.000018           9         2           dup
>> >   0.00    0.000010           1         9           close
>> >   0.00    0.000010           3         4           fcntl
>> >   0.00    0.000005           3         2           access
>> >   0.00    0.000002           2         1           mmap
>> >   0.00    0.000002           0        26           sched_yield
>> >   0.00    0.000001           0         8           recvfrom
>> >   0.00    0.000001           1         2           readlink
>> > ------ ----------- ----------- --------- --------- ----------------
>> > 100.00   46.930865                 51887     13349 total
>> >
>> > $ sudo strace -f -p 5972
>> > strace: Process 5972 attached with 86 threads
>> > [pid  7524] restart_syscall(<... resuming interrupted futex ...>
>> > <unfinished ...>
>> > [pid  7523] restart_syscall(<... resuming interrupted futex ...>
>> > <unfinished ...>
>> > [pid  7511] epoll_wait(142,  <unfinished ...>
>> > [pid  7471] epoll_wait(160,  <unfinished ...>
>> > [pid  7266] restart_syscall(<... resuming interrupted futex ...>
>> > <unfinished ...>
>> > [pid  7267] epoll_wait(151,  <unfinished ...>
>> > [pid  7264] futex(0x7fcbd8f3eb98, FUTEX_WAIT_PRIVATE, 0, NULL
>> <unfinished
>> > ...>
>> > [pid  7265] epoll_wait(157,  <unfinished ...>
>> > [pid  6311] futex(0x7fcabc8c42f8, FUTEX_WAIT_PRIVATE, 0, NULL
>> <unfinished
>> > ...>
>> > [pid  6310] futex(0x7fcab843c998, FUTEX_WAIT_PRIVATE, 0, NULL
>> <unfinished
>> > ...>
>> > [pid  6281] epoll_wait(154,  <unfinished ...>
>> > [pid  6280] futex(0x7fca68001d7c, FUTEX_WAIT_PRIVATE, 0, NULL
>> <unfinished
>> > ...>
>> > [pid  6278] futex(0x7fca74001d78, FUTEX_WAIT_PRIVATE, 0, NULL
>> <unfinished
>> > ...>
>> > [pid  6277] accept(136,  <unfinished ...>
>> > [pid  6272] futex(0x7fca7c001d7c, FUTEX_WAIT_PRIVATE, 0, NULL
>> <unfinished
>> > ...>
>> > [pid  6271] epoll_wait(208,  <unfinished ...>
>> > [pid  6270] epoll_wait(205,  <unfinished ...>
>> > [pid  6262] epoll_wait(199,  <unfinished ...>
>> > [pid  6261] epoll_wait(196,  <unfinished ...>
>> > [pid  6258] epoll_wait(193,  <unfinished ...>
>> > [pid  6269] epoll_wait(202,  <unfinished ...>
>> > [pid  6257] epoll_wait(190,  <unfinished ...>
>> > [pid  6256] epoll_wait(187,  <unfinished ...>
>> > [pid  6255] epoll_wait(184,  <unfinished ...>
>> > [pid  6253] epoll_wait(178,  <unfinished ...>
>> > [pid  6250] restart_syscall(<... resuming interrupted futex ...>
>> > <unfinished ...>
>> > [pid  6252] epoll_wait(175,  <unfinished ...>
>> > [pid  6251] epoll_wait(172,  <unfinished ...>
>> > [pid  6254] epoll_wait(181,  <unfinished ...>
>> > [pid  6246] futex(0x7fcbd8f3f378, FUTEX_WAIT_PRIVATE, 0, NULL
>> <unfinished
>> > ...>
>> > [pid  6238] restart_syscall(<... resuming interrupted futex ...>
>> > <unfinished ...>
>> > [pid  6237] restart_syscall(<... resuming interrupted futex ...>
>> > <unfinished ...>
>> > [pid  6235] futex(0x7fcbd83d97b8, FUTEX_WAIT_PRIVATE, 0, NULL
>> <unfinished
>> > ...>
>> > [pid  6234] restart_syscall(<... resuming interrupted futex ...>
>> > <unfinished ...>
>> > [pid  6233] restart_syscall(<... resuming interrupted futex ...>
>> > <unfinished ...>
>> > [pid  6232] futex(0x7fcad8002608, FUTEX_WAIT_PRIVATE, 0, NULL
>> <unfinished
>> > ...>
>> > [pid  6231] futex(0x7fcbd89425ac, FUTEX_WAIT_PRIVATE, 0, NULL
>> <unfinished
>> > ...>
>> > [pid  6049] futex(0x7fcaf4002388, FUTEX_WAIT_PRIVATE, 0, NULL
>> <unfinished
>> > ...>
>> > [pid  6230] epoll_wait(134,  <unfinished ...>
>> > [pid  6048] futex(0x7fcbd8db9ac8, FUTEX_WAIT_PRIVATE, 0, NULL
>> <unfinished
>> > ...>
>> > [pid  6047] epoll_wait(130,  <unfinished ...>
>> > [pid  6046] restart_syscall(<... resuming interrupted futex ...>
>> > <unfinished ...>
>> > [pid  6045] restart_syscall(<... resuming interrupted futex ...>
>> > <unfinished ...>
>> >
>> > JVM args:
>> >
>> >
>> >    -
>> >
>> -DSTOP.KEY=solrrocks-DSTOP.PORT=7983-Djetty.home=/opt/solr/server-Djetty.port=8983-Dlog4j.configurationFile=file:/opt/solr/server/resources/log4j2.xml-Dsolr.data.home=-Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf-Dsolr.install.dir=/opt/solr-Dsolr.jetty.https.port=8983-Dsolr.log.dir=/opt/solr/example/cloud/node1/solr/../logs-Dsolr.log.muteconsole-Dsolr.solr.home=/opt/solr/example/cloud/node1/solr-Duser.timezone=UTC-DzkClientTimeout=15000-DzkHost=localhost:2181-XX:+CMSParallelRemarkEnabled-XX:+CMSScavengeBeforeRemark-XX:+ParallelRefProcEnabled-XX:+PrintGCApplicationStoppedTime-XX:+PrintGCDateStamps-XX:+PrintGCDetails-XX:+PrintGCTimeStamps-XX:+PrintHeapAtGC-XX:+PrintTenuringDistribution-XX:+UseCMSInitiatingOccupancyOnly-XX:+UseConcMarkSweepGC-XX:+UseGCLogFileRotation-XX:-OmitStackTraceInFastThrow-XX:CMSInitiatingOccupancyFraction=50-XX:CMSMaxAbortablePrecleanTime=6000-XX:ConcGCThreads=4-XX:GCLogFileSize=20M-XX:MaxTenuringThreshold=8-XX:NewRatio=3-XX:NumberOfGCLogFiles=9-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh
>> >    8983
>> >
>> >
>> /opt/solr/example/cloud/node1/solr/../logs-XX:ParallelGCThreads=4-XX:PretenureSizeThreshold=64m-XX:SurvivorRatio=4-XX:TargetSurvivorRatio=90-Xloggc:/opt/solr/example/cloud/node1/solr/../logs/solr_gc.log-Xms512m-Xmx15g-Xss256k-verbose:gc
>> >
>> > Java Thread Dumb:
>> > name
>> > cpuTime / userTime
>> > process reaper (135)
>> > java.util.concurrent.SynchronousQueue$TransferStack@70b8631c
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.parkNanos
>> (LockSupport.java:215)
>> > • java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill
>> > (SynchronousQueue.java:460)
>> > • java.util.concurrent.SynchronousQueue$TransferStack.transfer
>> > (SynchronousQueue.java:362)
>> > • java.util.concurrent.SynchronousQueue.poll (SynchronousQueue.java:941)
>> > • java.util.concurrent.ThreadPoolExecutor.getTask
>> > (ThreadPoolExecutor.java:1073)
>> > • java.util.concurrent.ThreadPoolExecutor.runWorker
>> > (ThreadPoolExecutor.java:1134)
>> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
>> > (ThreadPoolExecutor.java:624)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 17.2219ms
>> > 0.0000ms
>> > qtp1571967156-129 (129)
>> > 36.0159ms
>> > 0.0000ms
>> > process reaper (117)
>> > java.util.concurrent.SynchronousQueue$TransferStack@70b8631c
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.parkNanos
>> (LockSupport.java:215)
>> > • java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill
>> > (SynchronousQueue.java:460)
>> > • java.util.concurrent.SynchronousQueue$TransferStack.transfer
>> > (SynchronousQueue.java:362)
>> > • java.util.concurrent.SynchronousQueue.poll (SynchronousQueue.java:941)
>> > • java.util.concurrent.ThreadPoolExecutor.getTask
>> > (ThreadPoolExecutor.java:1073)
>> > • java.util.concurrent.ThreadPoolExecutor.runWorker
>> > (ThreadPoolExecutor.java:1134)
>> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
>> > (ThreadPoolExecutor.java:624)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 19.2751ms
>> > 0.0000ms
>> > qtp1571967156-116 (116)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 1.5766ms
>> > 0.0000ms
>> > qtp1571967156-115 (115)
>> > • sun.management.ThreadImpl.getThreadInfo1 (Native Method)
>> > • sun.management.ThreadImpl.getThreadInfo (ThreadImpl.java:176)
>> > • org.apache.solr.handler.admin.ThreadDumpHandler.handleRequestBody
>> > (ThreadDumpHandler.java:71)
>> > • org.apache.solr.handler.RequestHandlerBase.handleRequest
>> > (RequestHandlerBase.java:199)
>> > • org.apache.solr.handler.admin.InfoHandler.handle (InfoHandler.java:91)
>> > • org.apache.solr.handler.admin.InfoHandler.handleRequestBody
>> > (InfoHandler.java:81)
>> > • org.apache.solr.handler.RequestHandlerBase.handleRequest
>> > (RequestHandlerBase.java:199)
>> > • org.apache.solr.servlet.HttpSolrCall.handleAdmin
>> (HttpSolrCall.java:734)
>> > • org.apache.solr.servlet.HttpSolrCall.handleAdminRequest
>> > (HttpSolrCall.java:715)
>> > • org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:496)
>> > • org.apache.solr.servlet.SolrDispatchFilter.doFilter
>> > (SolrDispatchFilter.java:377)
>> > • org.apache.solr.servlet.SolrDispatchFilter.doFilter
>> > (SolrDispatchFilter.java:323)
>> > • org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter
>> > (ServletHandler.java:1634)
>> > • org.eclipse.jetty.servlet.ServletHandler.doHandle
>> > (ServletHandler.java:533)
>> > • org.eclipse.jetty.server.handler.ScopedHandler.handle
>> > (ScopedHandler.java:146)
>> > • org.eclipse.jetty.security.SecurityHandler.handle
>> > (SecurityHandler.java:548)
>> > • org.eclipse.jetty.server.handler.HandlerWrapper.handle
>> > (HandlerWrapper.java:132)
>> > • org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
>> > (ScopedHandler.java:257)
>> > • org.eclipse.jetty.server.session.SessionHandler.doHandle
>> > (SessionHandler.java:1595)
>> > • org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
>> > (ScopedHandler.java:255)
>> > • org.eclipse.jetty.server.handler.ContextHandler.doHandle
>> > (ContextHandler.java:1317)
>> > • org.eclipse.jetty.server.handler.ScopedHandler.nextScope
>> > (ScopedHandler.java:203)
>> > • org.eclipse.jetty.servlet.ServletHandler.doScope
>> > (ServletHandler.java:473)
>> > • org.eclipse.jetty.server.session.SessionHandler.doScope
>> > (SessionHandler.java:1564)
>> > • org.eclipse.jetty.server.handler.ScopedHandler.nextScope
>> > (ScopedHandler.java:201)
>> > • org.eclipse.jetty.server.handler.ContextHandler.doScope
>> > (ContextHandler.java:1219)
>> > • org.eclipse.jetty.server.handler.ScopedHandler.handle
>> > (ScopedHandler.java:144)
>> > • org.eclipse.jetty.server.handler.ContextHandlerCollection.handle
>> > (ContextHandlerCollection.java:219)
>> > • org.eclipse.jetty.server.handler.HandlerCollection.handle
>> > (HandlerCollection.java:126)
>> > • org.eclipse.jetty.server.handler.HandlerWrapper.handle
>> > (HandlerWrapper.java:132)
>> > • org.eclipse.jetty.rewrite.handler.RewriteHandler.handle
>> > (RewriteHandler.java:335)
>> > • org.eclipse.jetty.server.handler.HandlerWrapper.handle
>> > (HandlerWrapper.java:132)
>> > • org.eclipse.jetty.server.Server.handle (Server.java:531)
>> > • org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:352)
>> > • org.eclipse.jetty.server.HttpConnection.onFillable
>> > (HttpConnection.java:260)
>> > • org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded
>> > (AbstractConnection.java:281)
>> > • org.eclipse.jetty.io.FillInterest.fillable (FillInterest.java:102)
>> > • org.eclipse.jetty.io.ChannelEndPoint$2.run (ChannelEndPoint.java:118)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask
>> > (EatWhatYouKill.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:310)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 81.0773ms
>> > 70.0000ms
>> > qtp1571967156-114 (114)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 14.7759ms
>> > 10.0000ms
>> > qtp1571967156-113 (113)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 1.7541ms
>> > 0.0000ms
>> > qtp1571967156-112 (112)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 28.2561ms
>> > 10.0000ms
>> > qtp1571967156-109 (109)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 20.5324ms
>> > 10.0000ms
>> > qtp1571967156-107 (107)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 13.4180ms
>> > 0.0000ms
>> > qtp1571967156-90 (90)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 108.5163ms
>> > 100.0000ms
>> > Scheduler-842741472 (89)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@27022346
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.parkNanos
>> (LockSupport.java:215)
>> > •
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
>> > (AbstractQueuedSynchronizer.java:2078)
>> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
>> > (ScheduledThreadPoolExecutor.java:1093)
>> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
>> > (ScheduledThreadPoolExecutor.java:809)
>> > • java.util.concurrent.ThreadPoolExecutor.getTask
>> > (ThreadPoolExecutor.java:1074)
>> > • java.util.concurrent.ThreadPoolExecutor.runWorker
>> > (ThreadPoolExecutor.java:1134)
>> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
>> > (ThreadPoolExecutor.java:624)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 5.4794ms
>> > 0.0000ms
>> > qtp1571967156-88 (88)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 38.6417ms
>> > 10.0000ms
>> > SolrRrdBackendFactory-11-thread-2 (87)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7813101a
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.parkNanos
>> (LockSupport.java:215)
>> > •
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
>> > (AbstractQueuedSynchronizer.java:2078)
>> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
>> > (ScheduledThreadPoolExecutor.java:1093)
>> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
>> > (ScheduledThreadPoolExecutor.java:809)
>> > • java.util.concurrent.ThreadPoolExecutor.getTask
>> > (ThreadPoolExecutor.java:1074)
>> > • java.util.concurrent.ThreadPoolExecutor.runWorker
>> > (ThreadPoolExecutor.java:1134)
>> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
>> > (ThreadPoolExecutor.java:624)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 3.3288ms
>> > 0.0000ms
>> > searcherExecutor-15-thread-1 (83)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4004f6b2
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
>> > •
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
>> > (AbstractQueuedSynchronizer.java:2039)
>> > • java.util.concurrent.LinkedBlockingQueue.take
>> > (LinkedBlockingQueue.java:442)
>> > • java.util.concurrent.ThreadPoolExecutor.getTask
>> > (ThreadPoolExecutor.java:1074)
>> > • java.util.concurrent.ThreadPoolExecutor.runWorker
>> > (ThreadPoolExecutor.java:1134)
>> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
>> > (ThreadPoolExecutor.java:624)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 8.9663ms
>> > 0.0000ms
>> > searcherExecutor-14-thread-1 (84)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7397cd2d
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
>> > •
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
>> > (AbstractQueuedSynchronizer.java:2039)
>> > • java.util.concurrent.LinkedBlockingQueue.take
>> > (LinkedBlockingQueue.java:442)
>> > • java.util.concurrent.ThreadPoolExecutor.getTask
>> > (ThreadPoolExecutor.java:1074)
>> > • java.util.concurrent.ThreadPoolExecutor.runWorker
>> > (ThreadPoolExecutor.java:1134)
>> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
>> > (ThreadPoolExecutor.java:624)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 10.0677ms
>> > 0.0000ms
>> > qtp1571967156-79 (79)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 110.6471ms
>> > 90.0000ms
>> > qtp1571967156-78-acceptor-3@6942c593-ServerConnector
>> > @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (78)
>> > java.lang.Object@4539d80a
>> > • sun.nio.ch.ServerSocketChannelImpl.accept
>> > (ServerSocketChannelImpl.java:234)
>> > • org.eclipse.jetty.server.ServerConnector.accept
>> > (ServerConnector.java:369)
>> > • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
>> > (AbstractConnector.java:639)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 4.9023ms
>> > 0.0000ms
>> > qtp1571967156-77-acceptor-2@26493f4e-ServerConnector
>> > @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (77)
>> > java.lang.Object@4539d80a
>> > • sun.nio.ch.ServerSocketChannelImpl.accept
>> > (ServerSocketChannelImpl.java:234)
>> > • org.eclipse.jetty.server.ServerConnector.accept
>> > (ServerConnector.java:369)
>> > • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
>> > (AbstractConnector.java:639)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 5.0835ms
>> > 0.0000ms
>> > qtp1571967156-76-acceptor-1@292b8c3e-ServerConnector
>> > @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (76)
>> > • sun.nio.ch.ServerSocketChannelImpl.accept0 (Native Method)
>> > • sun.nio.ch.ServerSocketChannelImpl.accept
>> > (ServerSocketChannelImpl.java:422)
>> > • sun.nio.ch.ServerSocketChannelImpl.accept
>> > (ServerSocketChannelImpl.java:250)
>> > • org.eclipse.jetty.server.ServerConnector.accept
>> > (ServerConnector.java:369)
>> > • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
>> > (AbstractConnector.java:639)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 5.6738ms
>> > 0.0000ms
>> > qtp1571967156-75-acceptor-0@10c4f96f-ServerConnector
>> > @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (75)
>> > java.lang.Object@4539d80a
>> > • sun.nio.ch.ServerSocketChannelImpl.accept
>> > (ServerSocketChannelImpl.java:234)
>> > • org.eclipse.jetty.server.ServerConnector.accept
>> > (ServerConnector.java:369)
>> > • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
>> > (AbstractConnector.java:639)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 4.9762ms
>> > 0.0000ms
>> > qtp1571967156-74 (74)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.parkNanos
>> (LockSupport.java:215)
>> > •
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
>> > (AbstractQueuedSynchronizer.java:2078)
>> > • org.eclipse.jetty.util.BlockingArrayQueue.poll
>> > (BlockingArrayQueue.java:392)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
>> > (QueuedThreadPool.java:653)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
>> > (QueuedThreadPool.java:48)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:717)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 4.6754ms
>> > 0.0000ms
>> > qtp1571967156-73 (73)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.parkNanos
>> (LockSupport.java:215)
>> > •
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
>> > (AbstractQueuedSynchronizer.java:2078)
>> > • org.eclipse.jetty.util.BlockingArrayQueue.poll
>> > (BlockingArrayQueue.java:392)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
>> > (QueuedThreadPool.java:653)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
>> > (QueuedThreadPool.java:48)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:717)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 6.0771ms
>> > 0.0000ms
>> > qtp1571967156-72 (72)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 28.1516ms
>> > 10.0000ms
>> > qtp1571967156-71 (71)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 31.7190ms
>> > 10.0000ms
>> > qtp1571967156-70 (70)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 22.8869ms
>> > 0.0000ms
>> > qtp1571967156-69 (69)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.parkNanos
>> (LockSupport.java:215)
>> > •
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
>> > (AbstractQueuedSynchronizer.java:2078)
>> > • org.eclipse.jetty.util.BlockingArrayQueue.poll
>> > (BlockingArrayQueue.java:392)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
>> > (QueuedThreadPool.java:653)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
>> > (QueuedThreadPool.java:48)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:717)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 5.6719ms
>> > 0.0000ms
>> > qtp1571967156-68 (68)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 7.0843ms
>> > 0.0000ms
>> > qtp1571967156-67 (67)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 25.2077ms
>> > 10.0000ms
>> > qtp1571967156-66 (66)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 59.3490ms
>> > 40.0000ms
>> > qtp1571967156-65 (65)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 77.9195ms
>> > 60.0000ms
>> > qtp1571967156-64 (64)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 44.4205ms
>> > 30.0000ms
>> > qtp1571967156-63 (63)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
>> > (ManagedSelector.java:396)
>> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
>> > (ManagedSelector.java:333)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
>> > (EatWhatYouKill.java:357)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
>> > (EatWhatYouKill.java:181)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
>> > (EatWhatYouKill.java:168)
>> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
>> > (EatWhatYouKill.java:126)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:366)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 129.4624ms
>> > 90.0000ms
>> > qtp1571967156-62 (62)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@421090bf
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.parkNanos
>> (LockSupport.java:215)
>> > •
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
>> > (AbstractQueuedSynchronizer.java:2163)
>> > •
>> >
>> >
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait
>> > (ReservedThreadExecutor.java:292)
>> > •
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
>> > (ReservedThreadExecutor.java:357)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
>> > (QueuedThreadPool.java:762)
>> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
>> > (QueuedThreadPool.java:680)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 51.2605ms
>> > 30.0000ms
>> > DestroyJavaVM (61)
>> > •
>> > 2002.8222ms
>> > 1850.0000ms
>> > Thread-13 (58)
>> > java.lang.Object@f2f13c1
>> > • java.lang.Object.wait (Native Method)
>> > • java.lang.Object.wait (Object.java:502)
>> > • org.apache.solr.core.CloserThread.run (CoreContainer.java:1850)
>> > 0.7894ms
>> > 0.0000ms
>> > MetricsHistoryHandler-12-thread-1 (55)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@45d73a6
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.parkNanos
>> (LockSupport.java:215)
>> > •
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
>> > (AbstractQueuedSynchronizer.java:2078)
>> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
>> > (ScheduledThreadPoolExecutor.java:1093)
>> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
>> > (ScheduledThreadPoolExecutor.java:809)
>> > • java.util.concurrent.ThreadPoolExecutor.getTask
>> > (ThreadPoolExecutor.java:1074)
>> > • java.util.concurrent.ThreadPoolExecutor.runWorker
>> > (ThreadPoolExecutor.java:1134)
>> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
>> > (ThreadPoolExecutor.java:624)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 348.4029ms
>> > 310.0000ms
>> > SolrRrdBackendFactory-11-thread-1 (54)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7813101a
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
>> > •
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
>> > (AbstractQueuedSynchronizer.java:2039)
>> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
>> > (ScheduledThreadPoolExecutor.java:1088)
>> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
>> > (ScheduledThreadPoolExecutor.java:809)
>> > • java.util.concurrent.ThreadPoolExecutor.getTask
>> > (ThreadPoolExecutor.java:1074)
>> > • java.util.concurrent.ThreadPoolExecutor.runWorker
>> > (ThreadPoolExecutor.java:1134)
>> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
>> > (ThreadPoolExecutor.java:624)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 5.7281ms
>> > 0.0000ms
>> >
>> >
>> OverseerAutoScalingTriggerThread-72060990031069185-192.168.1.40:8983_solr-n_0000001018
>> > (52)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@75fd9961
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
>> > •
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
>> > (AbstractQueuedSynchronizer.java:2039)
>> > • org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run
>> > (OverseerTriggerThread.java:189)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 58.1405ms
>> > 50.0000ms
>> >
>> >
>> OverseerCollectionConfigSetProcessor-72060990031069185-192.168.1.40:8983_solr-n_0000001018
>> > (51)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@55c10b44
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.parkNanos
>> (LockSupport.java:215)
>> > •
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
>> > (AbstractQueuedSynchronizer.java:2078)
>> > • org.apache.solr.cloud.ZkDistributedQueue.peekElements
>> > (ZkDistributedQueue.java:463)
>> > • org.apache.solr.cloud.OverseerTaskQueue.peekTopN
>> > (OverseerTaskQueue.java:243)
>> > • org.apache.solr.cloud.OverseerTaskProcessor.run
>> > (OverseerTaskProcessor.java:233)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 220.6397ms
>> > 170.0000ms
>> >
>> OverseerStateUpdate-72060990031069185-192.168.1.40:8983_solr-n_0000001018
>> > (47)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2287ab26
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.parkNanos
>> (LockSupport.java:215)
>> > •
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
>> > (AbstractQueuedSynchronizer.java:2078)
>> > • org.apache.solr.cloud.ZkDistributedQueue.peekElements
>> > (ZkDistributedQueue.java:463)
>> > • org.apache.solr.cloud.Overseer$ClusterStateUpdater.run
>> > (Overseer.java:207)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 189.5649ms
>> > 140.0000ms
>> > zkConnectionManagerCallback-13-thread-1 (50)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4da83c93
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
>> > •
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
>> > (AbstractQueuedSynchronizer.java:2039)
>> > • java.util.concurrent.LinkedBlockingQueue.take
>> > (LinkedBlockingQueue.java:442)
>> > • java.util.concurrent.ThreadPoolExecutor.getTask
>> > (ThreadPoolExecutor.java:1074)
>> > • java.util.concurrent.ThreadPoolExecutor.runWorker
>> > (ThreadPoolExecutor.java:1134)
>> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
>> > (ThreadPoolExecutor.java:624)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 1.1550ms
>> > 0.0000ms
>> > main-EventThread (49)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@540792d6
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
>> > •
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
>> > (AbstractQueuedSynchronizer.java:2039)
>> > • java.util.concurrent.LinkedBlockingQueue.take
>> > (LinkedBlockingQueue.java:442)
>> > • org.apache.zookeeper.ClientCnxn$EventThread.run (ClientCnxn.java:502)
>> > 2.6375ms
>> > 0.0000ms
>> > main-SendThread(localhost:2181) (48)
>> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
>> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
>> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
>> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
>> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
>> > • org.apache.zookeeper.ClientCnxnSocketNIO.doTransport
>> > (ClientCnxnSocketNIO.java:349)
>> > • org.apache.zookeeper.ClientCnxn$SendThread.run (ClientCnxn.java:1144)
>> > 52.9289ms
>> > 10.0000ms
>> > zkConnectionManagerCallback-6-thread-1 (44)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@9c13839
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
>> > •
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
>> > (AbstractQueuedSynchronizer.java:2039)
>> > • java.util.concurrent.LinkedBlockingQueue.take
>> > (LinkedBlockingQueue.java:442)
>> > • java.util.concurrent.ThreadPoolExecutor.getTask
>> > (ThreadPoolExecutor.java:1074)
>> > • java.util.concurrent.ThreadPoolExecutor.runWorker
>> > (ThreadPoolExecutor.java:1134)
>> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
>> > (ThreadPoolExecutor.java:624)
>> > • java.lang.Thread.run (Thread.java:748)
>> > 1.1128ms
>> > 0.0000ms
>> > main-EventThread (43)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16806cbc
>> > • sun.misc.Unsafe.park (Native Method)
>> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
>> > •
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
>> > (AbstractQueuedSynchronizer.java:2039)
>> > • java.util.concurrent.LinkedBlockingQueue.take
>> > (LinkedBlockingQueue.java:442)
>> > • org.apache.zookeeper.ClientCnxn$EventThread.run (ClientCnxn.java:502)
>> >
>> >
>> > --
>> >
>> > --
>> > Daniel Mantovani
>> >
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>
>
> --
>
> --
> Daniel Mantovani
>
>

--

--
Daniel Mantovani