[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-11) - Build # 3183 - Unstable!

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-11) - Build # 3183 - Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3183/
Java: 64bit/jdk-11 -XX:+UseCompressedOops -XX:+UseSerialGC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TriLevelCompositeIdRoutingTest

Error Message:
42 threads leaked from SUITE scope at org.apache.solr.cloud.TriLevelCompositeIdRoutingTest:     1) Thread[id=8092, name=qtp892050889-8092, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)         at java.base@11/java.lang.Thread.run(Thread.java:834)    2) Thread[id=8180, name=MetricsHistoryHandler-3080-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@11/java.lang.Thread.run(Thread.java:834)    3) Thread[id=8139, name=Scheduler-668999846, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@11/java.lang.Thread.run(Thread.java:834)    4) Thread[id=8102, name=qtp602784508-8102-acceptor-0@8735b79-ServerConnector@4eeabd27{SSL,[ssl, http/1.1]}{127.0.0.1:46405}, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)         at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.base@11/java.lang.Thread.run(Thread.java:834)    5) Thread[id=8090, name=qtp892050889-8090, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)         at java.base@11/java.lang.Thread.run(Thread.java:834)    6) Thread[id=8194, name=Scheduler-1422058016, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@11/java.lang.Thread.run(Thread.java:834)    7) Thread[id=8159, name=MetricsHistoryHandler-3073-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@11/java.lang.Thread.run(Thread.java:834)    8) Thread[id=8182, name=coreContainerWorkExecutor-3067-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)         at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)         at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@11/java.lang.Thread.run(Thread.java:834)    9) Thread[id=8138, name=qtp1916476557-8138, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)         at java.base@11/java.lang.Thread.run(Thread.java:834)   10) Thread[id=8128, name=coreContainerWorkExecutor-3056-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)         at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)         at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@11/java.lang.Thread.run(Thread.java:834)   11) Thread[id=8135, name=qtp1916476557-8135-acceptor-0@212a007c-ServerConnector@1d8d0d83{SSL,[ssl, http/1.1]}{127.0.0.1:45679}, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)         at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.base@11/java.lang.Thread.run(Thread.java:834)   12) Thread[id=8160, name=Thread-1463, state=WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/java.lang.Object.wait(Native Method)         at java.base@11/java.lang.Object.wait(Object.java:328)         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1892)   13) Thread[id=8093, name=Scheduler-1143194803, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@11/java.lang.Thread.run(Thread.java:834)   14) Thread[id=8104, name=qtp602784508-8104, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)         at java.base@11/java.lang.Thread.run(Thread.java:834)   15) Thread[id=8125, name=SolrRrdBackendFactory-3063-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@11/java.lang.Thread.run(Thread.java:834)   16) Thread[id=8195, name=Scheduler-1657850055, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@11/java.lang.Thread.run(Thread.java:834)   17) Thread[id=8136, name=qtp1916476557-8136, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)         at java.base@11/java.lang.Thread.run(Thread.java:834)   18) Thread[id=8089, name=qtp892050889-8089-acceptor-0@2b8c8138-ServerConnector@4b364da1{SSL,[ssl, http/1.1]}{127.0.0.1:35227}, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)         at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.base@11/java.lang.Thread.run(Thread.java:834)   19) Thread[id=8131, name=qtp1916476557-8131, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/sun.nio.ch.EPoll.wait(Native Method)         at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.base@11/java.lang.Thread.run(Thread.java:834)   20) Thread[id=8132, name=qtp1916476557-8132, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/sun.nio.ch.EPoll.wait(Native Method)         at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.base@11/java.lang.Thread.run(Thread.java:834)   21) Thread[id=8179, name=SolrRrdBackendFactory-3079-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@11/java.lang.Thread.run(Thread.java:834)   22) Thread[id=8085, name=qtp892050889-8085, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/sun.nio.ch.EPoll.wait(Native Method)         at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.base@11/java.lang.Thread.run(Thread.java:834)   23) Thread[id=8134, name=qtp1916476557-8134, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/sun.nio.ch.EPoll.wait(Native Method)         at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.base@11/java.lang.Thread.run(Thread.java:834)   24) Thread[id=8100, name=qtp602784508-8100, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/sun.nio.ch.EPoll.wait(Native Method)         at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.base@11/java.lang.Thread.run(Thread.java:834)   25) Thread[id=8088, name=qtp892050889-8088, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/sun.nio.ch.EPoll.wait(Native Method)         at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.base@11/java.lang.Thread.run(Thread.java:834)   26) Thread[id=8193, name=Scheduler-1947637746, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@11/java.lang.Thread.run(Thread.java:834)   27) Thread[id=8099, name=qtp602784508-8099, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/sun.nio.ch.EPoll.wait(Native Method)         at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.base@11/java.lang.Thread.run(Thread.java:834)   28) Thread[id=8098, name=qtp602784508-8098, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/sun.nio.ch.EPoll.wait(Native Method)         at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.base@11/java.lang.Thread.run(Thread.java:834)   29) Thread[id=8086, name=qtp892050889-8086, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/sun.nio.ch.EPoll.wait(Native Method)         at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.base@11/java.lang.Thread.run(Thread.java:834)   30) Thread[id=8133, name=qtp1916476557-8133, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/sun.nio.ch.EPoll.wait(Native Method)         at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.base@11/java.lang.Thread.run(Thread.java:834)   31) Thread[id=8103, name=qtp602784508-8103, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)         at java.base@11/java.lang.Thread.run(Thread.java:834)   32) Thread[id=8126, name=MetricsHistoryHandler-3064-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@11/java.lang.Thread.run(Thread.java:834)   33) Thread[id=8137, name=qtp1916476557-8137, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)         at java.base@11/java.lang.Thread.run(Thread.java:834)   34) Thread[id=8127, name=Thread-1454, state=WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/java.lang.Object.wait(Native Method)         at java.base@11/java.lang.Object.wait(Object.java:328)         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1892)   35) Thread[id=8158, name=SolrRrdBackendFactory-3072-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@11/java.lang.Thread.run(Thread.java:834)   36) Thread[id=8181, name=Thread-1464, state=WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/java.lang.Object.wait(Native Method)         at java.base@11/java.lang.Object.wait(Object.java:328)         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1892)   37) Thread[id=8087, name=qtp892050889-8087, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/sun.nio.ch.EPoll.wait(Native Method)         at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.base@11/java.lang.Thread.run(Thread.java:834)   38) Thread[id=8091, name=qtp892050889-8091, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)         at java.base@11/java.lang.Thread.run(Thread.java:834)   39) Thread[id=8161, name=coreContainerWorkExecutor-3058-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)         at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)         at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@11/java.lang.Thread.run(Thread.java:834)   40) Thread[id=8106, name=Scheduler-1331626504, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@11/java.lang.Thread.run(Thread.java:834)   41) Thread[id=8101, name=qtp602784508-8101, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/sun.nio.ch.EPoll.wait(Native Method)         at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.base@11/java.lang.Thread.run(Thread.java:834)   42) Thread[id=8105, name=qtp602784508-8105, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)         at java.base@11/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 42 threads leaked from SUITE scope at org.apache.solr.cloud.TriLevelCompositeIdRoutingTest:
   1) Thread[id=8092, name=qtp892050889-8092, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
   2) Thread[id=8180, name=MetricsHistoryHandler-3080-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
   3) Thread[id=8139, name=Scheduler-668999846, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
   4) Thread[id=8102, name=qtp602784508-8102-acceptor-0@8735b79-ServerConnector@4eeabd27{SSL,[ssl, http/1.1]}{127.0.0.1:46405}, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
        at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
   5) Thread[id=8090, name=qtp892050889-8090, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
   6) Thread[id=8194, name=Scheduler-1422058016, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
   7) Thread[id=8159, name=MetricsHistoryHandler-3073-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
   8) Thread[id=8182, name=coreContainerWorkExecutor-3067-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
   9) Thread[id=8138, name=qtp1916476557-8138, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  10) Thread[id=8128, name=coreContainerWorkExecutor-3056-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  11) Thread[id=8135, name=qtp1916476557-8135-acceptor-0@212a007c-ServerConnector@1d8d0d83{SSL,[ssl, http/1.1]}{127.0.0.1:45679}, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
        at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  12) Thread[id=8160, name=Thread-1463, state=WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/java.lang.Object.wait(Native Method)
        at java.base@11/java.lang.Object.wait(Object.java:328)
        at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1892)
  13) Thread[id=8093, name=Scheduler-1143194803, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  14) Thread[id=8104, name=qtp602784508-8104, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  15) Thread[id=8125, name=SolrRrdBackendFactory-3063-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  16) Thread[id=8195, name=Scheduler-1657850055, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  17) Thread[id=8136, name=qtp1916476557-8136, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  18) Thread[id=8089, name=qtp892050889-8089-acceptor-0@2b8c8138-ServerConnector@4b364da1{SSL,[ssl, http/1.1]}{127.0.0.1:35227}, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
        at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  19) Thread[id=8131, name=qtp1916476557-8131, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  20) Thread[id=8132, name=qtp1916476557-8132, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  21) Thread[id=8179, name=SolrRrdBackendFactory-3079-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  22) Thread[id=8085, name=qtp892050889-8085, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  23) Thread[id=8134, name=qtp1916476557-8134, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  24) Thread[id=8100, name=qtp602784508-8100, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  25) Thread[id=8088, name=qtp892050889-8088, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  26) Thread[id=8193, name=Scheduler-1947637746, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  27) Thread[id=8099, name=qtp602784508-8099, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  28) Thread[id=8098, name=qtp602784508-8098, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  29) Thread[id=8086, name=qtp892050889-8086, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  30) Thread[id=8133, name=qtp1916476557-8133, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  31) Thread[id=8103, name=qtp602784508-8103, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  32) Thread[id=8126, name=MetricsHistoryHandler-3064-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  33) Thread[id=8137, name=qtp1916476557-8137, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  34) Thread[id=8127, name=Thread-1454, state=WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/java.lang.Object.wait(Native Method)
        at java.base@11/java.lang.Object.wait(Object.java:328)
        at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1892)
  35) Thread[id=8158, name=SolrRrdBackendFactory-3072-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  36) Thread[id=8181, name=Thread-1464, state=WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/java.lang.Object.wait(Native Method)
        at java.base@11/java.lang.Object.wait(Object.java:328)
        at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1892)
  37) Thread[id=8087, name=qtp892050889-8087, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  38) Thread[id=8091, name=qtp892050889-8091, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  39) Thread[id=8161, name=coreContainerWorkExecutor-3058-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  40) Thread[id=8106, name=Scheduler-1331626504, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  41) Thread[id=8101, name=qtp602784508-8101, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
  42) Thread[id=8105, name=qtp602784508-8105, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
        at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
        at java.base@11/java.lang.Thread.run(Thread.java:834)
        at __randomizedtesting.SeedInfo.seed([1D4EE4E4107AD3F6]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TriLevelCompositeIdRoutingTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=8180, name=MetricsHistoryHandler-3080-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE

[...truncated too long message...]

take(ScheduledThreadPoolExecutor.java:1182)
   [junit4]    >         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
   [junit4]    >         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
   [junit4]    >         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
   [junit4]    >         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   [junit4]    >         at java.base@11/java.lang.Thread.run(Thread.java:834)
   [junit4]    >   24) Thread[id=8127, name=Thread-1454, state=WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
   [junit4]    >         at java.base@11/java.lang.Object.wait(Native Method)
   [junit4]    >         at java.base@11/java.lang.Object.wait(Object.java:328)
   [junit4]    >         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1892)
   [junit4]    >   25) Thread[id=8158, name=SolrRrdBackendFactory-3072-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
   [junit4]    >         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
   [junit4]    >         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
   [junit4]    >         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
   [junit4]    >         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
   [junit4]    >         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   [junit4]    >         at java.base@11/java.lang.Thread.run(Thread.java:834)
   [junit4]    >   26) Thread[id=8181, name=Thread-1464, state=WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
   [junit4]    >         at java.base@11/java.lang.Object.wait(Native Method)
   [junit4]    >         at java.base@11/java.lang.Object.wait(Object.java:328)
   [junit4]    >         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1892)
   [junit4]    >   27) Thread[id=8087, name=qtp892050889-8087, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]
   [junit4]    >         at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
   [junit4]    >         at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
   [junit4]    >         at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
   [junit4]    >         at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
   [junit4]    >         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
   [junit4]    >         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
   [junit4]    >         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
   [junit4]    >         at java.base@11/java.lang.Thread.run(Thread.java:834)
   [junit4]    >   28) Thread[id=8161, name=coreContainerWorkExecutor-3058-thread-1, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
   [junit4]    >         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
   [junit4]    >         at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
   [junit4]    >         at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
   [junit4]    >         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
   [junit4]    >         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
   [junit4]    >         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   [junit4]    >         at java.base@11/java.lang.Thread.run(Thread.java:834)
   [junit4]    >   29) Thread[id=8106, name=Scheduler-1331626504, state=TIMED_WAITING, group=TGRP-TriLevelCompositeIdRoutingTest]
   [junit4]    >         at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
   [junit4]    >         at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
   [junit4]    >         at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
   [junit4]    >         at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
   [junit4]    >         at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   [junit4]    >         at java.base@11/java.lang.Thread.run(Thread.java:834)
   [junit4]    >   30) Thread[id=8101, name=qtp602784508-8101, state=RUNNABLE, group=TGRP-TriLevelCompositeIdRoutingTest]
   [junit4]    >         at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
   [junit4]    >         at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
   [junit4]    >         at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
   [junit4]    >         at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
   [junit4]    >         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
   [junit4]    >         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
   [junit4]    >         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$150/0x0000000100304840.run(Unknown Source)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
   [junit4]    >         at java.base@11/java.lang.Thread.run(Thread.java:834)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([1D4EE4E4107AD3F6]:0)
   [junit4] Completed [218/836 (1!)] on J2 in 44.47s, 1 test, 3 errors <<< FAILURES!

[...truncated 44621 lines...]
[repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3183/consoleText

[repro] Revision: ec1d05ba960c2613ececa0c5cb20d0ca030d50bf

[repro] Ant options: "-Dargs=-XX:+UseCompressedOops -XX:+UseSerialGC"
[repro] Repro line:  ant test  -Dtestcase=TriLevelCompositeIdRoutingTest -Dtests.method=test -Dtests.seed=1D4EE4E4107AD3F6 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-PN -Dtests.timezone=Asia/Jayapura -Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=TriLevelCompositeIdRoutingTest -Dtests.seed=1D4EE4E4107AD3F6 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-PN -Dtests.timezone=Asia/Jayapura -Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr/core
[repro]       TriLevelCompositeIdRoutingTest
[repro] ant compile-test

[...truncated 3592 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 -Dtests.class="*.TriLevelCompositeIdRoutingTest" -Dtests.showOutput=onerror "-Dargs=-XX:+UseCompressedOops -XX:+UseSerialGC" -Dtests.seed=1D4EE4E4107AD3F6 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-PN -Dtests.timezone=Asia/Jayapura -Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[...truncated 75 lines...]
[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.TriLevelCompositeIdRoutingTest
[repro] Exiting with code 0

[...truncated 40 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]