[JENKINS] Lucene-Solr-NightlyTests-7.2 - Build # 6 - Still Failing

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.2 - Build # 6 - Still Failing

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.2/6/

13 tests failed.
FAILED:  org.apache.lucene.index.TestBinaryDocValuesUpdates.testTonsOfUpdates

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at __randomizedtesting.SeedInfo.seed([7C2475C2A17CDE9E:401ABC9435CF17C]:0)
        at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:78)
        at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:51)
        at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:164)
        at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:150)
        at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:141)
        at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addBinaryField(MemoryDocValuesConsumer.java:330)
        at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addBinaryField(MemoryDocValuesConsumer.java:303)
        at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addBinaryField(PerFieldDocValuesFormat.java:114)
        at org.apache.lucene.index.ReadersAndUpdates.handleBinaryDVUpdates(ReadersAndUpdates.java:542)
        at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:719)
        at org.apache.lucene.index.IndexWriter$ReaderPool.writeSomeDocValuesUpdates(IndexWriter.java:705)
        at org.apache.lucene.index.FrozenBufferedUpdates.apply(FrozenBufferedUpdates.java:331)
        at org.apache.lucene.index.DocumentsWriter$ResolveUpdatesEvent.process(DocumentsWriter.java:739)
        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5106)
        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5096)
        at org.apache.lucene.index.IndexWriter.updateDocValues(IndexWriter.java:1891)
        at org.apache.lucene.index.TestBinaryDocValuesUpdates.testTonsOfUpdates(TestBinaryDocValuesUpdates.java:1323)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestBinaryDocValuesUpdates

Error Message:
Captured an uncaught exception in thread: Thread[id=5193, name=Lucene Merge Thread #1, state=RUNNABLE, group=TGRP-TestBinaryDocValuesUpdates]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5193, name=Lucene Merge Thread #1, state=RUNNABLE, group=TGRP-TestBinaryDocValuesUpdates]
        at __randomizedtesting.SeedInfo.seed([7C2475C2A17CDE9E]:0)
Caused by: org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
        at __randomizedtesting.SeedInfo.seed([7C2475C2A17CDE9E]:0)
        at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
        at org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:345)
        at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:697)
        at org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:692)
        at org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5030)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4529)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083)
        at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at __randomizedtesting.SeedInfo.seed([7C2475C2A17CDE9E:401ABC9435CF17C]:0)
        at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:78)
        at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:51)
        at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:164)
        at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:150)
        at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:141)
        at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addBinaryField(MemoryDocValuesConsumer.java:330)
        at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addBinaryField(MemoryDocValuesConsumer.java:303)
        at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addBinaryField(PerFieldDocValuesFormat.java:114)
        at org.apache.lucene.index.ReadersAndUpdates.handleBinaryDVUpdates(ReadersAndUpdates.java:542)
        at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:719)
        at org.apache.lucene.index.IndexWriter$ReaderPool.writeSomeDocValuesUpdates(IndexWriter.java:705)
        at org.apache.lucene.index.FrozenBufferedUpdates.apply(FrozenBufferedUpdates.java:331)
        at org.apache.lucene.index.DocumentsWriter$ResolveUpdatesEvent.process(DocumentsWriter.java:739)
        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5106)
        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5096)
        at org.apache.lucene.index.IndexWriter.updateDocValues(IndexWriter.java:1891)
        at org.apache.lucene.index.TestBinaryDocValuesUpdates.testTonsOfUpdates(TestBinaryDocValuesUpdates.java:1323)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)


FAILED:  org.apache.lucene.index.TestIndexWriterOnVMError.testCheckpoint

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([7C2475C2A17CDE9E]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterOnVMError

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([7C2475C2A17CDE9E]:0)


FAILED:  org.apache.lucene.index.TestMixedDocValuesUpdates.testTonsOfUpdates

Error Message:
_2_i.fnm in dir=RAMDirectory@10e60665 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3fe50ce6

Stack Trace:
java.nio.file.NoSuchFileException: _2_i.fnm in dir=RAMDirectory@10e60665 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3fe50ce6
        at __randomizedtesting.SeedInfo.seed([7C2475C2A17CDE9E:401ABC9435CF17C]:0)
        at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:750)
        at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:119)
        at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
        at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:113)
        at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:190)
        at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:93)
        at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:688)
        at org.apache.lucene.index.IndexWriter$ReaderPool.writeSomeDocValuesUpdates(IndexWriter.java:705)
        at org.apache.lucene.index.FrozenBufferedUpdates.apply(FrozenBufferedUpdates.java:331)
        at org.apache.lucene.index.DocumentsWriter$ResolveUpdatesEvent.process(DocumentsWriter.java:739)
        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5106)
        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5096)
        at org.apache.lucene.index.IndexWriter.updateDocValues(IndexWriter.java:1891)
        at org.apache.lucene.index.TestMixedDocValuesUpdates.testTonsOfUpdates(TestMixedDocValuesUpdates.java:374)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.lucene.index.TestNumericDocValuesUpdates.testTonsOfUpdates

Error Message:
_1_s.fnm in dir=RAMDirectory@7b76ac36 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6437d882

Stack Trace:
java.nio.file.NoSuchFileException: _1_s.fnm in dir=RAMDirectory@7b76ac36 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6437d882
        at __randomizedtesting.SeedInfo.seed([7C2475C2A17CDE9E:401ABC9435CF17C]:0)
        at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:750)
        at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:119)
        at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
        at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:113)
        at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:190)
        at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:93)
        at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:688)
        at org.apache.lucene.index.IndexWriter$ReaderPool.writeSomeDocValuesUpdates(IndexWriter.java:705)
        at org.apache.lucene.index.FrozenBufferedUpdates.apply(FrozenBufferedUpdates.java:331)
        at org.apache.lucene.index.DocumentsWriter$ResolveUpdatesEvent.process(DocumentsWriter.java:739)
        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5106)
        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5096)
        at org.apache.lucene.index.IndexWriter.updateDocValues(IndexWriter.java:1891)
        at org.apache.lucene.index.TestNumericDocValuesUpdates.testTonsOfUpdates(TestNumericDocValuesUpdates.java:1557)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RollingRestartTest

Error Message:
398 threads leaked from SUITE scope at org.apache.solr.cloud.RollingRestartTest:     1) Thread[id=69913, name=zkCallback-4588-thread-3, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=70092, name=coreZkRegister-8086-thread-2, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=69613, name=Scheduler-1966782150, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=69935, name=zkCallback-4608-thread-3, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=70004, name=zkConnectionManagerCallback-4549-thread-1-processing-n:127.0.0.1:52101__xn%2Fmw-SendThread(127.0.0.1:47872), state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    6) Thread[id=69608, name=zkConnectionManagerCallback-4559-thread-1, state=WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=69592, name=qtp655047689-69592, state=RUNNABLE, group=TGRP-RollingRestartTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)    8) Thread[id=69922, name=zkCallback-4529-thread-2, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    9) Thread[id=69811, name=Connection evictor, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   10) Thread[id=69526, name=qtp274493890-69526, state=RUNNABLE, group=TGRP-RollingRestartTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   11) Thread[id=69737, name=org.eclipse.jetty.server.session.HashSessionManager@148587b6Timer, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   12) Thread[id=69855, name=Thread-51996, state=WAITING, group=TGRP-RollingRestartTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1717)   13) Thread[id=70075, name=coreZkRegister-7998-thread-1, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   14) Thread[id=69501, name=Connection evictor, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   15) Thread[id=69897, name=zkCallback-4638-thread-3, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   16) Thread[id=69981, name=zkConnectionManagerCallback-4589-thread-1-processing-n:127.0.0.1:39716__xn%2Fmw-EventThread, state=WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)   17) Thread[id=70086, name=coreZkRegister-8108-thread-2, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   18) Thread[id=69947, name=zkCallback-4638-thread-5, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   19) Thread[id=69661, name=qtp158363611-69661-acceptor-0@737432d4-ServerConnector@469daa96{SSL,[ssl, http/1.1]}{127.0.0.1:39415}, state=RUNNABLE, group=TGRP-RollingRestartTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   20) Thread[id=69731, name=qtp1102235373-69731-acceptor-0@4787617-ServerConnector@15d5bb2b{SSL,[ssl, http/1.1]}{127.0.0.1:57108}, state=RUNNABLE, group=TGRP-RollingRestartTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   21) Thread[id=70043, name=updateExecutor-4634-thread-2, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   22) Thread[id=69807, name=org.eclipse.jetty.server.session.HashSessionManager@13d25fe1Timer, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   23) Thread[id=69706, name=Connection evictor, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   24) Thread[id=69906, name=zkCallback-4520-thread-2, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   25) Thread[id=69462, name=qtp281352983-69462, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   26) Thread[id=69839, name=qtp29947028-69839, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   27) Thread[id=70035, name=updateExecutor-4584-thread-2, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   28) Thread[id=69465, name=qtp281352983-69465, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   29) Thread[id=69802, name=qtp1663321053-69802, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   30) Thread[id=70085, name=coreZkRegister-8108-thread-1, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   31) Thread[id=69820, name=Thread-51987, state=WAITING, group=TGRP-RollingRestartTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1717)   32) Thread[id=69719, name=watches-4590-thread-1, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   33) Thread[id=69847, name=Connection evictor, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   34) Thread[id=69464, name=qtp281352983-69464, state=RUNNABLE, group=TGRP-RollingRestartTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   35) Thread[id=70019, name=ScheduledTrigger-8137-thread-3, state=WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   36) Thread[id=70143, name=zkCallback-4568-thread-7, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   37) Thread[id=70002, name=OverseerThreadFactory-8139-thread-1, state=WAITING, group=Overseer collection creation process.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   38) Thread[id=70020, name=ScheduledTrigger-8137-thread-4, state=WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   39) Thread[id=69477, name=zkConnectionManagerCallback-4521-thread-1, state=WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   40) Thread[id=69684, name=watches-4580-thread-1, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   41) Thread[id=69998, name=zkConnectionManagerCallback-4569-thread-1-processing-n:127.0.0.1:57473__xn%2Fmw-SendThread(127.0.0.1:47872), state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)   42) Thread[id=69988, name=zkCallback-4588-thread-6, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   43) Thread[id=69928, name=zkCallback-4568-thread-2, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   44) Thread[id=69959, name=zkCallback-4618-thread-6, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   45) Thread[id=69994, name=zkCallback-4538-thread-6, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   46) Thread[id=69915, name=zkCallback-4548-thread-2, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   47) Thread[id=69566, name=Connection evictor, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   48) Thread[id=69914, name=zkCallback-4588-thread-4, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   49) Thread[id=69470, name=Connection evictor, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   50) Thread[id=69660, name=qtp158363611-69660, state=RUNNABLE, group=TGRP-RollingRestartTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)         at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   51) Thread[id=69630, name=qtp54748075-69630, state=RUNNABLE, group=TGRP-RollingRestartTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   52) Thread[id=69939, name=zkCallback-4578-thread-4, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   53) Thread[id=69573, name=zkConnectionManagerCallback-4549-thread-1, state=WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   54) Thread[id=69480, name=Thread-51897, state=WAITING, group=TGRP-RollingRestartTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1717)   55) Thread[id=69911, name=zkCallback-4538-thread-4, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   56) Thread[id=69858, name=Scheduler-1538003379, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   57) Thread[id=69686, name=searcherExecutor-8070-thread-1, state=WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   58) Thread[id=69963, name=zkConnectionManagerCallback-4619-thread-1-processing-n:127.0.0.1:53061__xn%2Fmw-SendThread(127.0.0.1:47872), state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)   59) Thread[id=70142, name=zkCallback-4628-thread-7, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   60) Thread[id=69859, name=watches-4630-thread-1, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   61) Thread[id=70049, name=qtp428857385-70049, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   62) Thread[id=69561, name=qtp1888312601-69561, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   63) Thread[id=69846, name=Connection evictor, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   64) Thread[id=70123, name=recoveryExecutor-4595-thread-2, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   65) Thread[id=69698, name=qtp682851457-69698, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   66) Thread[id=69777, name=Connection evictor, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   67) Thread[id=69990, name=zkCallback-4598-thread-6, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   68) Thread[id=69645, name=Thread-51942, state=WAITING, group=TGRP-RollingRestartTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1717)   69) Thread[id=69514, name=watches-4531-thread-1, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   70) Thread[id=70028, name=recoveryExecutor-4635-thread-1, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   71) Thread[id=69595, name=qtp655047689-69595, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   72) Thread[id=70137, name=zkCallback-4598-thread-7, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   73) Thread[id=69463, name=qtp281352983-69463, state=RUNNABLE, group=TGRP-RollingRestartTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   74) Thread[id=69559, name=qtp1888312601-69559, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   75) Thread[id=69764, name=qtp1432070361-69764, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   76) Thread[id=69602, name=Connection evictor, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   77) Thread[id=69748, name=zkConnectionManagerCallback-4599-thread-1, state=WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   78) Thread[id=69953, name=zkCallback-4608-thread-6, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   79) Thread[id=69574, name=zkCallback-4548-thread-1, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   80) Thread[id=69824, name=watches-4620-thread-1, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   81) Thread[id=69968, name=zkConnectionManagerCallback-4609-thread-1-processing-n:127.0.0.1:48500__xn%2Fmw-SendThread(127.0.0.1:47872), state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230)         at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1170)   82) Thread[id=70007, name=coreZkRegister-8097-thread-1, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   83) Thread[id=69776, name=Connection evictor, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   84) Thread[id=69488, name=Connection evictor, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   85) Thread[id=70057, name=coreZkRegister-8031-thread-2, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   86) Thread[id=69871, name=qtp610825330-69871-acceptor-0@63e9e57c-ServerConnector@5aeb155f{SSL,[ssl, http/1.1]}{127.0.0.1:40066}, state=RUNNABLE, group=TGRP-RollingRestartTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   87) Thread[id=69983, name=zkCallback-4548-thread-6, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   88) Thread[id=70061, name=coreZkRegister-8053-thread-2, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   89) Thread[id=69795, name=Connection evictor, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   90) Thread[id=69648, name=Scheduler-849976405, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   91) Thread[id=69918, name=zkCallback-4618-thread-2, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   92) Thread[id=69680, name=Thread-51951, state=WAITING, group=TGRP-RollingRestartTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1717)   93) Thread[id=69997, name=zkCallback-4520-thread-7, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   94) Thread[id=69531, name=Connection evictor, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   95) Thread[id=69830, name=Connection evictor, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   96) Thread[id=69976, name=zkConnectionManagerCallback-4629-thread-1-processing-n:127.0.0.1:37958__xn%2Fmw-EventThread, state=WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)   97) Thread[id=69649, name=watches-4570-thread-1, state=TIMED_WAITING, group=TGRP-RollingRestartTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.Synchro

[...truncated too long message...]

-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/byte-buddy-1.6.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/commons-math3-3.6.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/hadoop-common-2.7.4-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/hadoop-hdfs-2.7.4-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/hadoop-minikdc-2.7.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/jetty-sslengine-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/mockito-core-2.6.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/test-lib/objenesis-2.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/lucene/build/analysis/icu/lucene-analyzers-icu-7.2.1-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/lucene/build/backward-codecs/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/contrib/analysis-extras/lib/icu4j-59.1.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/usr/local/asfpackages/java/jdk1.8.0_144/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.3.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/temp/junit4-J2-20180110_163600_1462499360877468773229.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/temp/junit4-J2-20180110_163600_14672030802385912353.suites -stdin
   [junit4] ERROR: JVM J2 ended with an exception: Forked process returned with error code: 1. Very likely a JVM crash.  See process stdout at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/temp/junit4-J2-20180110_163600_1467143193336791818694.sysout See process stderr at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/temp/junit4-J2-20180110_163600_14612114683115598507.syserr
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1519)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4] at java.lang.Thread.run(Thread.java:748)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/build.xml:843: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/build.xml:779: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build.xml:262: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/common-build.xml:556: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/lucene/common-build.xml:1512: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/lucene/common-build.xml:1038: At least one slave process threw an exception, first: Forked process returned with error code: 1. Very likely a JVM crash.  See process stdout at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/temp/junit4-J2-20180110_163600_1467143193336791818694.sysout See process stderr at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/temp/junit4-J2-20180110_163600_14612114683115598507.syserr

Total time: 1109 minutes 51 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[Fast Archiver] No artifacts from Lucene-Solr-NightlyTests-7.2 #4 to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.2 - Build # 7 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.2/7/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest

Error Message:
6 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest:     1) Thread[id=21907, name=zkCallback-281-thread-4, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=7242, name=zkConnectionManagerCallback-282-thread-1, state=WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=21906, name=zkCallback-281-thread-3, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=7241, name=StoppableIndexingThread-EventThread, state=WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    5) Thread[id=21849, name=zkCallback-281-thread-2, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=7240, name=StoppableIndexingThread-SendThread(127.0.0.1:37307), state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 6 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest:
   1) Thread[id=21907, name=zkCallback-281-thread-4, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=7242, name=zkConnectionManagerCallback-282-thread-1, state=WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=21906, name=zkCallback-281-thread-3, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=7241, name=StoppableIndexingThread-EventThread, state=WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   5) Thread[id=21849, name=zkCallback-281-thread-2, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=7240, name=StoppableIndexingThread-SendThread(127.0.0.1:37307), state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
        at __randomizedtesting.SeedInfo.seed([9910A7879E9A95D8]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=21907, name=zkCallback-281-thread-4, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=7242, name=zkConnectionManagerCallback-282-thread-1, state=WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=21906, name=zkCallback-281-thread-3, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=21849, name=zkCallback-281-thread-2, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=7240, name=StoppableIndexingThread-SendThread(127.0.0.1:37307), state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=21907, name=zkCallback-281-thread-4, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=7242, name=zkConnectionManagerCallback-282-thread-1, state=WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=21906, name=zkCallback-281-thread-3, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=21849, name=zkCallback-281-thread-2, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=7240, name=StoppableIndexingThread-SendThread(127.0.0.1:37307), state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
        at __randomizedtesting.SeedInfo.seed([9910A7879E9A95D8]:0)


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:
Expected numSlices=5 numReplicas=1 but found DocCollection(solrj_collection4//collections/solrj_collection4/state.json/26)={   "pullReplicas":"0",   "replicationFactor":"1",   "shards":{     "shard1":{       "range":"80000000-b332ffff",       "state":"active",       "replicas":{"core_node4":{           "dataDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node4/data/",           "base_url":"http://127.0.0.1:44676",           "node_name":"127.0.0.1:44676_",           "type":"NRT",           "ulogDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node4/data/tlog",           "core":"solrj_collection4_shard1_replica_n1",           "shared_storage":"true",           "state":"active",           "leader":"true"}}},     "shard2":{       "range":"b3330000-e665ffff",       "state":"active",       "replicas":{"core_node6":{           "dataDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node6/data/",           "base_url":"http://127.0.0.1:39324",           "node_name":"127.0.0.1:39324_",           "type":"NRT",           "ulogDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node6/data/tlog",           "core":"solrj_collection4_shard2_replica_n2",           "shared_storage":"true",           "state":"active",           "leader":"true"}}},     "shard3":{       "range":"e6660000-1998ffff",       "state":"active",       "replicas":{"core_node8":{           "core":"solrj_collection4_shard3_replica_n3",           "base_url":"http://127.0.0.1:39324",           "node_name":"127.0.0.1:39324_",           "state":"down",           "type":"NRT"}}},     "shard4":{       "range":"19990000-4ccbffff",       "state":"active",       "replicas":{"core_node9":{           "dataDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node9/data/",           "base_url":"http://127.0.0.1:39324",           "node_name":"127.0.0.1:39324_",           "type":"NRT",           "ulogDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node9/data/tlog",           "core":"solrj_collection4_shard4_replica_n5",           "shared_storage":"true",           "state":"active",           "leader":"true"}}},     "shard5":{       "range":"4ccc0000-7fffffff",       "state":"active",       "replicas":{"core_node10":{           "dataDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node10/data/",           "base_url":"http://127.0.0.1:35783",           "node_name":"127.0.0.1:35783_",           "type":"NRT",           "ulogDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node10/data/tlog",           "core":"solrj_collection4_shard5_replica_n7",           "shared_storage":"true",           "state":"active",           "leader":"true"}}}},   "router":{     "field":"text",     "name":"compositeId"},   "maxShardsPerNode":"5",   "autoAddReplicas":"true",   "nrtReplicas":"1",   "tlogReplicas":"0"} with /live_nodes: [127.0.0.1:35783_, 127.0.0.1:37030_, 127.0.0.1:39324_, 127.0.0.1:44676_, 127.0.0.1:45543_]

Stack Trace:
java.lang.AssertionError: Expected numSlices=5 numReplicas=1 but found DocCollection(solrj_collection4//collections/solrj_collection4/state.json/26)={
  "pullReplicas":"0",
  "replicationFactor":"1",
  "shards":{
    "shard1":{
      "range":"80000000-b332ffff",
      "state":"active",
      "replicas":{"core_node4":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node4/data/",
          "base_url":"http://127.0.0.1:44676",
          "node_name":"127.0.0.1:44676_",
          "type":"NRT",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node4/data/tlog",
          "core":"solrj_collection4_shard1_replica_n1",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}},
    "shard2":{
      "range":"b3330000-e665ffff",
      "state":"active",
      "replicas":{"core_node6":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node6/data/",
          "base_url":"http://127.0.0.1:39324",
          "node_name":"127.0.0.1:39324_",
          "type":"NRT",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node6/data/tlog",
          "core":"solrj_collection4_shard2_replica_n2",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}},
    "shard3":{
      "range":"e6660000-1998ffff",
      "state":"active",
      "replicas":{"core_node8":{
          "core":"solrj_collection4_shard3_replica_n3",
          "base_url":"http://127.0.0.1:39324",
          "node_name":"127.0.0.1:39324_",
          "state":"down",
          "type":"NRT"}}},
    "shard4":{
      "range":"19990000-4ccbffff",
      "state":"active",
      "replicas":{"core_node9":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node9/data/",
          "base_url":"http://127.0.0.1:39324",
          "node_name":"127.0.0.1:39324_",
          "type":"NRT",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node9/data/tlog",
          "core":"solrj_collection4_shard4_replica_n5",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}},
    "shard5":{
      "range":"4ccc0000-7fffffff",
      "state":"active",
      "replicas":{"core_node10":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node10/data/",
          "base_url":"http://127.0.0.1:35783",
          "node_name":"127.0.0.1:35783_",
          "type":"NRT",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node10/data/tlog",
          "core":"solrj_collection4_shard5_replica_n7",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}}},
  "router":{
    "field":"text",
    "name":"compositeId"},
  "maxShardsPerNode":"5",
  "autoAddReplicas":"true",
  "nrtReplicas":"1",
  "tlogReplicas":"0"} with /live_nodes: [127.0.0.1:35783_, 127.0.0.1:37030_, 127.0.0.1:39324_, 127.0.0.1:44676_, 127.0.0.1:45543_]
        at __randomizedtesting.SeedInfo.seed([9910A7879E9A95D8:1144985D3066F820]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.assertSliceAndReplicaCount(SharedFSAutoReplicaFailoverTest.java:413)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:258)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:143)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([9910A7879E9A95D8:1144985D3066F820]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12782 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9910A7879E9A95D8-001/init-core-data-001
   [junit4]   2> 711758 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 711829 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 711831 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 711831 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 711831 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 714665 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 717692 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 718522 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
   [junit4]   2> 718625 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 720331 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 720552 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_lucene2.us.west_apache_org_44700_hdfs____t3id7l/webapp
   [junit4]   2> 723338 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.m.log Started HttpServer2$[hidden email]:44700
   [junit4]   2> 727690 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 727707 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 727768 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_35157_datanode____.v9lj8o/webapp
   [junit4]   2> 728728 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35157
   [junit4]   2> 731555 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 731673 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 731919 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_42686_datanode____.l6qlyb/webapp
   [junit4]   2> 734090 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9910A7879E9A95D8]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42686
   [junit4]   2> 739532 ERROR (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9910A7879E9A95D8-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9910A7879E9A95D8-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:45565) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 739532 ERROR (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9910A7879E9A95D8-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9910A7879E9A95D8-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:45565) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 742261 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x99128259307f: from storage DS-ba42bff3-d699-4ac2-82df-0770f674b18c node DatanodeRegistration(127.0.0.1:35176, datanodeUuid=a8d89727-bc24-4f32-9887-88a5b58818ab, infoPort=41079, infoSecurePort=0, ipcPort=40686, storageInfo=lv=-56;cid=testClusterID;nsid=80848523;c=0), blocks: 0, hasStaleStorage: true, processing time: 96 msecs
   [junit4]   2> 742267 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x99128259307f: from storage DS-524ad0d4-63d5-4c10-85c9-4ca2cb6a8a90 node DatanodeRegistration(127.0.0.1:35176, datanodeUuid=a8d89727-bc24-4f32-9887-88a5b58818ab, infoPort=41079, infoSecurePort=0, ipcPort=40686, storageInfo=lv=-56;cid=testClusterID;nsid=80848523;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 742332 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x9912836daccc: from storage DS-e93660bb-91fc-4726-8271-131628bad3aa node DatanodeRegistration(127.0.0.1:38952, datanodeUuid=d9fc1e54-c960-45a2-bbcb-83a34b0edaf0, infoPort=41281, infoSecurePort=0, ipcPort=44198, storageInfo=lv=-56;cid=testClusterID;nsid=80848523;c=0), blocks: 0, hasStaleStorage: true, processing time: 49 msecs
   [junit4]   2> 742332 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x9912836daccc: from storage DS-49925226-0105-45d2-9c56-1972abb141a1 node DatanodeRegistration(127.0.0.1:38952, datanodeUuid=d9fc1e54-c960-45a2-bbcb-83a34b0edaf0, infoPort=41281, infoSecurePort=0, ipcPort=44198, storageInfo=lv=-56;cid=testClusterID;nsid=80848523;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 745250 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 745287 INFO  (Thread-514) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 745287 INFO  (Thread-514) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 745336 ERROR (Thread-514) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 745375 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.ZkTestServer start zk server on port:37307
   [junit4]   2> 745429 INFO  (zkConnectionManagerCallback-192-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 745510 INFO  (zkConnectionManagerCallback-194-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 745566 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 745568 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 745570 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 745583 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 745596 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 745597 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 745598 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 745628 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 745629 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 745631 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 745632 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 745634 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 746170 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 746184 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1c39b0ec{/,null,AVAILABLE}
   [junit4]   2> 746216 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@76d99ab8{HTTP/1.1,[http/1.1]}{127.0.0.1:42720}
   [junit4]   2> 746216 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.e.j.s.Server Started @756989ms
   [junit4]   2> 746216 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:45565/hdfs__lucene2-us-west.apache.org_45565__home_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.2_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9910A7879E9A95D8-001_tempDir-002_control_data, hostContext=/, hostPort=42720, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9910A7879E9A95D8-001/control-001/cores}
   [junit4]   2> 746216 ERROR (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 746216 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.2.1
   [junit4]   2> 746216 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 746216 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 746216 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-11T23:39:36.879Z
   [junit4]   2> 746249 INFO  (zkConnectionManagerCallback-196-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 746250 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 746251 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9910A7879E9A95D8-001/control-001/solr.xml
   [junit4]   2> 746255 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 746255 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 746273 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@27ab6af0, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 746288 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37307/solr
   [junit4]   2> 746330 INFO  (zkConnectionManagerCallback-200-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 746395 INFO  (zkConnectionManagerCallback-202-thread-1-processing-n:127.0.0.1:42720_) [n:127.0.0.1:42720_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 746665 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [n:127.0.0.1:42720_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 746678 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [n:127.0.0.1:42720_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42720_
   [junit4]   2> 746679 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [n:127.0.0.1:42720_    ] o.a.s.c.Overseer Overseer (id=99333831128907780-127.0.0.1:42720_-n_0000000000) starting
   [junit4]   2> 746758 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [n:127.0.0.1:42720_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42720_
   [junit4]   2> 746790 INFO  (zkCallback-201-thread-1-processing-n:127.0.0.1:42720_) [n:127.0.0.1:42720_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 747065 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [n:127.0.0.1:42720_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@27ab6af0
   [junit4]   2> 747121 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [n:127.0.0.1:42720_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@27ab6af0
   [junit4]   2> 747122 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [n:127.0.0.1:42720_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@27ab6af0
   [junit4]   2> 747123 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [n:127.0.0.1:42720_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9910A7879E9A95D8-001/control-001/cores
   [junit4]   2> 747231 INFO  (zkConnectionManagerCallback-208-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 747232 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 747233 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37307/solr ready
   [junit4]   2> 747274 INFO  (qtp56662695-1323) [n:127.0.0.1:42720_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:42720_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 747277 INFO  (OverseerThreadFactory-337-thread-1-processing-n:127.0.0.1:42720_) [n:127.0.0.1:42720_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 747449 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 747450 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 747562 INFO  (zkCallback-201-thread-1-processing-n:127.0.0.1:42720_) [n:127.0.0.1:42720_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 748525 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.2.1
   [junit4]   2> 748651 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 749362 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 749489 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 749490 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@27ab6af0
   [junit4]   2> 749636 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:45565/solr_hdfs_home
   [junit4]   2> 749636 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 749636 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 749637 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9910A7879E9A95D8-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[hdfs://lucene2-us-west.apache.org:45565/solr_hdfs_home/control_collection/core_node2/data/]
   [junit4]   2> 749670 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:45565/solr_hdfs_home/control_collection/core_node2/data/snapshot_metadata
   [junit4]   2> 749782 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 749782 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 749783 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 752265 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 752282 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:45565/solr_hdfs_home/control_collection/core_node2/data
   [junit4]   2> 752556 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:45565/solr_hdfs_home/control_collection/core_node2/data/index
   [junit4]   2> 752575 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 752575 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 752576 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 752629 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 752630 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=16, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8471231011159144]
   [junit4]   2> 753900 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38952 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-49925226-0105-45d2-9c56-1972abb141a1:NORMAL:127.0.0.1:38952|RBW], ReplicaUC[[DISK]DS-524ad0d4-63d5-4c10-85c9-4ca2cb6a8a90:NORMAL:127.0.0.1:35176|RBW]]} size 69
   [junit4]   2> 753901 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35176 is added to blk_1073741825_1001 size 69
   [junit4]   2> 754441 WARN  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 754653 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 754653 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 754653 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 754709 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
   [junit4]   2> 754709 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 754758 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=32, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 754924 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@45e1a2a0[control_collection_shard1_replica_n1] main]
   [junit4]   2> 754937 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 754938 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 754940 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 754944 INFO  (searcherExecutor-340-thread-1-processing-n:127.0.0.1:42720_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@45e1a2a0[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 754960 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1589341308171845632
   [junit4]   2> 754977 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 754977 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 754978 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:42720/control_collection_shard1_replica_n1/
   [junit4]   2> 754978 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 754978 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:42720/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 754978 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 754989 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:42720/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 754990 INFO  (zkCallback-201-thread-1-processing-n:127.0.0.1:42720_) [n:127.0.0.1:42720_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 755040 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 755042 INFO  (qtp56662695-1325) [n:127.0.0.1:42720_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=7592
   [junit4]   2> 755055 INFO  (qtp56662695-1323) [n:127.0.0.1:42720_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 755143 INFO  (zkCallback-201-thread-2-processing-n:127.0.0.1:42720_) [n:127.0.0.1:42720_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 755342 INFO  (OverseerCollectionConfigSetProcessor-99333831128907780-127.0.0.1:42720_-n_0000000000) [n:127.0.0.1:42720_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 756055 INFO  (qtp56662695-1323) [n:127.0.0.1:42720_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:42720_&wt=javabin&version=2} status=0 QTime=8781
   [junit4]   2> 756077 INFO  (zkConnectionManagerCallback-212-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 756078 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 756079 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37307/solr ready
   [junit4]   2> 756079 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 756080 INFO  (qtp56662695-1324) [n:127.0.0.1:42720_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=3&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 756104 INFO  (OverseerThreadFactory-337-thread-2-processing-n:127.0.0.1:42720_) [n:127.0.0.1:42720_    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 756105 WARN  (OverseerThreadFactory-337-thread-2-processing-n:127.0.0.1:42720_) [n:127.0.0.1:42720_    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 756356 INFO  (qtp56662695-1324) [n:127.0.0.1:42720_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 756356 INFO  (qtp56662695-1324) [n:127.0.0.1:42720_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=3&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=275
   [junit4]   2> 756666 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9910A7879E9A95D8-001/shard-1-001 of type NRT
   [junit4]   2> 756667 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 756675 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4a8a743e{/,null,AVAILABLE}
   [junit4]   2> 756676 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7047e76a{HTTP/1.1,[http/1.1]}{127.0.0.1:44639}
   [junit4]   2> 756676 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.e.j.s.Server Started @767449ms
   [junit4]   2> 756676 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:45565/hdfs__lucene2-us-west.apache.org_45565__home_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.2_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9910A7879E9A95D8-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=44639, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9910A7879E9A95D8-001/shard-1-001/cores}
   [junit4]   2> 756676 ERROR (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 756676 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.2.1
   [junit4]   2> 756676 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 756676 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 756676 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-11T23:39:47.339Z
   [junit4]   2> 756694 INFO  (zkConnectionManagerCallback-214-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 756757 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 756757 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9910A7879E9A95D8-001/shard-1-001/solr.xml
   [junit4]   2> 756774 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 756774 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 756775 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@27ab6af0, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 756793 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37307/solr
   [junit4]   2> 756838 INFO  (zkConnectionManagerCallback-218-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 756931 INFO  (zkConnectionManagerCallback-220-thread-1-processing-n:127.0.0.1:44639_) [n:127.0.0.1:44639_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 756954 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [n:127.0.0.1:44639_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 756973 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [n:127.0.0.1:44639_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 756992 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [n:127.0.0.1:44639_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44639_
   [junit4]   2> 756993 INFO  (zkCallback-201-thread-2-processing-n:127.0.0.1:42720_) [n:127.0.0.1:42720_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 757081 INFO  (zkCallback-211-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 757199 INFO  (zkCallback-219-thread-1-processing-n:127.0.0.1:44639_) [n:127.0.0.1:44639_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 757344 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [n:127.0.0.1:44639_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@27ab6af0
   [junit4]   2> 757377 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [n:127.0.0.1:44639_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@27ab6af0
   [junit4]   2> 757377 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [n:127.0.0.1:44639_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@27ab6af0
   [junit4]   2> 757379 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [n:127.0.0.1:44639_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9910A7879E9A95D8-001/shard-1-001/cores
   [junit4]   2> 757467 INFO  (qtp56662695-1320) [n:127.0.0.1:42720_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:44639_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 757469 INFO  (OverseerCollectionConfigSetProcessor-99333831128907780-127.0.0.1:42720_-n_0000000000) [n:127.0.0.1:42720_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 757479 INFO  (OverseerThreadFactory-337-thread-3-processing-n:127.0.0.1:42720_) [n:127.0.0.1:42720_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:44639_ for creating new replica
   [junit4]   2> 757561 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n61&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 757562 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 757698 INFO  (zkCallback-219-thread-1-processing-n:127.0.0.1:44639_) [n:127.0.0.1:44639_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 758820 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.2.1
   [junit4]   2> 759042 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.s.IndexSchema [collection1_shard2_replica_n61] Schema name=test
   [junit4]   2> 759553 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 759668 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n61' using configuration from collection collection1, trusted=true
   [junit4]   2> 759721 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n61' (registry 'solr.core.collection1.shard2.replica_n61') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@27ab6af0
   [junit4]   2> 759721 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:45565/solr_hdfs_home
   [junit4]   2> 759727 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 759727 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 759727 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SolrCore [[collection1_shard2_replica_n61] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9910A7879E9A95D8-001/shard-1-001/cores/collection1_shard2_replica_n61], dataDir=[hdfs://lucene2-us-west.apache.org:45565/solr_hdfs_home/collection1/core_node62/data/]
   [junit4]   2> 759728 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:45565/solr_hdfs_home/collection1/core_node62/data/snapshot_metadata
   [junit4]   2> 759769 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 759769 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 759769 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 759883 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 759952 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:45565/solr_hdfs_home/collection1/core_node62/data
   [junit4]   2> 760035 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:45565/solr_hdfs_home/collection1/core_node62/data/index
   [junit4]   2> 760083 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 760083 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 760083 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 760152 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 760152 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=16, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8471231011159144]
   [junit4]   2> 760449 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35176 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-49925226-0105-45d2-9c56-1972abb141a1:NORMAL:127.0.0.1:38952|RBW], ReplicaUC[[DISK]DS-524ad0d4-63d5-4c10-85c9-4ca2cb6a8a90:NORMAL:127.0.0.1:35176|FINALIZED]]} size 0
   [junit4]   2> 760517 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38952 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-524ad0d4-63d5-4c10-85c9-4ca2cb6a8a90:NORMAL:127.0.0.1:35176|FINALIZED], ReplicaUC[[DISK]DS-e93660bb-91fc-4726-8271-131628bad3aa:NORMAL:127.0.0.1:38952|FINALIZED]]} size 0
   [junit4]   2> 760650 WARN  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 760948 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 760948 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 760948 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 761048 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
   [junit4]   2> 761048 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 761069 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=32, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 761217 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.s.SolrIndexSearcher Opening [Searcher@40cca79e[collection1_shard2_replica_n61] main]
   [junit4]   2> 761258 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 761259 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 761259 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 761260 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1589341314794651648
   [junit4]   2> 761263 INFO  (searcherExecutor-351-thread-1-processing-n:127.0.0.1:44639_ x:collection1_shard2_replica_n61 s:shard2 c:collection1) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SolrCore [collection1_shard2_replica_n61] Registered new searcher Searcher@40cca79e[collection1_shard2_replica_n61] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 761276 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 761276 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 761276 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:44639/collection1_shard2_replica_n61/
   [junit4]   2> 761276 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 761276 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy http://127.0.0.1:44639/collection1_shard2_replica_n61/ has no replicas
   [junit4]   2> 761276 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 761290 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:44639/collection1_shard2_replica_n61/ shard2
   [junit4]   2> 761303 INFO  (zkCallback-219-thread-1-processing-n:127.0.0.1:44639_) [n:127.0.0.1:44639_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 761341 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 761343 INFO  (qtp1268555709-1381) [n:127.0.0.1:44639_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n61&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3781
   [junit4]   2> 761363 INFO  (qtp56662695-1320) [n:127.0.0.1:42720_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:44639_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=3896
   [junit4]   2> 761444 INFO  (zkCallback-219-thread-1-processing-n:127.0.0.1:44639_) [n:127.0.0.1:44639_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 761576 INFO  (OverseerCollectionConfigSetProcessor-99333831128907780-127.0.0.1:42720_-n_0000000000) [n:127.0.0.1:42720_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 762875 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9910A7879E9A95D8]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkin

[...truncated too long message...]

         "shared_storage":"true",
   [junit4]    >           "state":"active",
   [junit4]    >           "leader":"true"}}},
   [junit4]    >     "shard2":{
   [junit4]    >       "range":"b3330000-e665ffff",
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{"core_node6":{
   [junit4]    >           "dataDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node6/data/",
   [junit4]    >           "base_url":"http://127.0.0.1:39324",
   [junit4]    >           "node_name":"127.0.0.1:39324_",
   [junit4]    >           "type":"NRT",
   [junit4]    >           "ulogDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node6/data/tlog",
   [junit4]    >           "core":"solrj_collection4_shard2_replica_n2",
   [junit4]    >           "shared_storage":"true",
   [junit4]    >           "state":"active",
   [junit4]    >           "leader":"true"}}},
   [junit4]    >     "shard3":{
   [junit4]    >       "range":"e6660000-1998ffff",
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{"core_node8":{
   [junit4]    >           "core":"solrj_collection4_shard3_replica_n3",
   [junit4]    >           "base_url":"http://127.0.0.1:39324",
   [junit4]    >           "node_name":"127.0.0.1:39324_",
   [junit4]    >           "state":"down",
   [junit4]    >           "type":"NRT"}}},
   [junit4]    >     "shard4":{
   [junit4]    >       "range":"19990000-4ccbffff",
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{"core_node9":{
   [junit4]    >           "dataDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node9/data/",
   [junit4]    >           "base_url":"http://127.0.0.1:39324",
   [junit4]    >           "node_name":"127.0.0.1:39324_",
   [junit4]    >           "type":"NRT",
   [junit4]    >           "ulogDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node9/data/tlog",
   [junit4]    >           "core":"solrj_collection4_shard4_replica_n5",
   [junit4]    >           "shared_storage":"true",
   [junit4]    >           "state":"active",
   [junit4]    >           "leader":"true"}}},
   [junit4]    >     "shard5":{
   [junit4]    >       "range":"4ccc0000-7fffffff",
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{"core_node10":{
   [junit4]    >           "dataDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node10/data/",
   [junit4]    >           "base_url":"http://127.0.0.1:35783",
   [junit4]    >           "node_name":"127.0.0.1:35783_",
   [junit4]    >           "type":"NRT",
   [junit4]    >           "ulogDir":"hdfs://lucene2-us-west.apache.org:40497/solr_hdfs_home/solrj_collection4/core_node10/data/tlog",
   [junit4]    >           "core":"solrj_collection4_shard5_replica_n7",
   [junit4]    >           "shared_storage":"true",
   [junit4]    >           "state":"active",
   [junit4]    >           "leader":"true"}}}},
   [junit4]    >   "router":{
   [junit4]    >     "field":"text",
   [junit4]    >     "name":"compositeId"},
   [junit4]    >   "maxShardsPerNode":"5",
   [junit4]    >   "autoAddReplicas":"true",
   [junit4]    >   "nrtReplicas":"1",
   [junit4]    >   "tlogReplicas":"0"} with /live_nodes: [127.0.0.1:35783_, 127.0.0.1:37030_, 127.0.0.1:39324_, 127.0.0.1:44676_, 127.0.0.1:45543_]
   [junit4]    > at __randomizedtesting.SeedInfo.seed([9910A7879E9A95D8:1144985D3066F820]:0)
   [junit4]    > at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.assertSliceAndReplicaCount(SharedFSAutoReplicaFailoverTest.java:413)
   [junit4]    > at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:258)
   [junit4]    > at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:143)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2573049 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9910A7879E9A95D8]-worker) [    ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 2573057 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9910A7879E9A95D8]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 2573158 WARN  (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9910A7879E9A95D8-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9910A7879E9A95D8-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:40497) [    ] o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager interrupted
   [junit4]   2> 2573159 WARN  (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9910A7879E9A95D8-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9910A7879E9A95D8-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:40497) [    ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-690830785-127.0.0.1-1515714419858 (Datanode Uuid 7756d6c1-7286-4a23-8353-898ab2fca652) service to lucene2-us-west.apache.org/127.0.0.1:40497
   [junit4]   2> 2573160 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9910A7879E9A95D8]-worker) [    ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 2573166 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9910A7879E9A95D8]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 2573267 WARN  (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9910A7879E9A95D8-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9910A7879E9A95D8-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:40497) [    ] o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager interrupted
   [junit4]   2> 2573268 WARN  (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9910A7879E9A95D8-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9910A7879E9A95D8-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:40497) [    ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-690830785-127.0.0.1-1515714419858 (Datanode Uuid 54b45156-79ba-49a4-b11d-07575d5c78cc) service to lucene2-us-west.apache.org/127.0.0.1:40497
   [junit4]   2> 2573289 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[9910A7879E9A95D8]-worker) [    ] o.m.log Stopped HttpServer2$[hidden email]:0
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_9910A7879E9A95D8-001
   [junit4]   2> Jan 12, 2018 12:10:03 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 35 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), text=Lucene50(blocksize=128)}, docValues:{_version_=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1819, maxMBSortInHeap=5.830647428904523, sim=RandomSimilarity(queryNorm=true): {}, locale=sr, timezone=Europe/Ljubljana
   [junit4]   2> NOTE: Linux 4.4.0-104-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=5,free=173611792,total=526909440
   [junit4]   2> NOTE: All tests run in this JVM: [SolrCloudExampleTest, TestFreeTextSuggestions, TestLFUCache, UninvertDocValuesMergePolicyTest, TestDistributedGrouping, TestCloudDeleteByQuery, TestCharFilters, TestConfigSets, DistributedFacetPivotSmallTest, BlockDirectoryTest, TestFuzzyAnalyzedSuggestions, AddBlockUpdateTest, TestRequestForwarding, TestSystemIdResolver, TestCloudPivotFacet, OverseerStatusTest, GraphQueryTest, TestDistribDocBasedVersion, EnumFieldTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, SpellPossibilityIteratorTest, BadComponentTest, UnloadDistributedZkTest, TestLegacyTerms, TestOrdValues, IndexBasedSpellCheckerTest, TriLevelCompositeIdRoutingTest, TestLMDirichletSimilarityFactory, HLLUtilTest, RequiredFieldsTest, TestMiniSolrCloudClusterSSL, TestLeaderInitiatedRecoveryThread, TestDynamicLoading, SortSpecParsingTest, SharedFSAutoReplicaFailoverTest]
   [junit4] Completed [650/751 (2!)] on J2 in 1394.71s, 1 test, 1 failure <<< FAILURES!

[...truncated 47623 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.2 - Build # 8 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.2/8/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
6 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:     1) Thread[id=23176, name=zkConnectionManagerCallback-3721-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=23175, name=StoppableIndexingThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    3) Thread[id=23174, name=StoppableIndexingThread-SendThread(127.0.0.1:41803), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    4) Thread[id=23265, name=zkCallback-3720-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=23177, name=zkCallback-3720-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=23266, name=zkCallback-3720-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 6 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:
   1) Thread[id=23176, name=zkConnectionManagerCallback-3721-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=23175, name=StoppableIndexingThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   3) Thread[id=23174, name=StoppableIndexingThread-SendThread(127.0.0.1:41803), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   4) Thread[id=23265, name=zkCallback-3720-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=23177, name=zkCallback-3720-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=23266, name=zkCallback-3720-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([5F2751DD4FCACFEF]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=23176, name=zkConnectionManagerCallback-3721-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=23174, name=StoppableIndexingThread-SendThread(127.0.0.1:41803), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    3) Thread[id=23265, name=zkCallback-3720-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=23177, name=zkCallback-3720-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=23266, name=zkCallback-3720-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=23176, name=zkConnectionManagerCallback-3721-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=23174, name=StoppableIndexingThread-SendThread(127.0.0.1:41803), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   3) Thread[id=23265, name=zkCallback-3720-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=23177, name=zkCallback-3720-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=23266, name=zkCallback-3720-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([5F2751DD4FCACFEF]:0)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([5F2751DD4FCACFEF:D7736E07E136A217]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.MultiThreadedOCPTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([5F2751DD4FCACFEF:D7736E07E136A217]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.MultiThreadedOCPTest.testFillWorkQueue(MultiThreadedOCPTest.java:112)
        at org.apache.solr.cloud.MultiThreadedOCPTest.test(MultiThreadedOCPTest.java:67)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12449 lines...]
   [junit4] Suite: org.apache.solr.cloud.MultiThreadedOCPTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/init-core-data-001
   [junit4]   2> 507951 WARN  (SUITE-MultiThreadedOCPTest-seed#[5F2751DD4FCACFEF]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 507951 INFO  (SUITE-MultiThreadedOCPTest-seed#[5F2751DD4FCACFEF]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 507952 INFO  (SUITE-MultiThreadedOCPTest-seed#[5F2751DD4FCACFEF]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 508005 INFO  (SUITE-MultiThreadedOCPTest-seed#[5F2751DD4FCACFEF]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 508005 INFO  (SUITE-MultiThreadedOCPTest-seed#[5F2751DD4FCACFEF]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /cg_igk/ti
   [junit4]   2> 508113 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 508132 INFO  (Thread-691) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 508132 INFO  (Thread-691) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 508154 ERROR (Thread-691) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 508241 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.ZkTestServer start zk server on port:44410
   [junit4]   2> 508298 INFO  (zkConnectionManagerCallback-444-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 508387 INFO  (zkConnectionManagerCallback-446-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 508390 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 508440 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 508440 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 508441 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 508442 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 508442 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 508443 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 508460 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 508513 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 508513 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 508514 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 508606 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x160eeef0ed60001, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 508626 INFO  (zkConnectionManagerCallback-449-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 508643 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 509079 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 509082 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1ed2bf00{/cg_igk/ti,null,AVAILABLE}
   [junit4]   2> 509163 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5259c33d{SSL,[ssl, http/1.1]}{127.0.0.1:40559}
   [junit4]   2> 509163 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.e.j.s.Server Started @518882ms
   [junit4]   2> 509163 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/tempDir-001/control/data, hostContext=/cg_igk/ti, hostPort=40559, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/control-001/cores}
   [junit4]   2> 509163 ERROR (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 509163 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.2.1
   [junit4]   2> 509163 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 509163 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 509163 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-13T09:52:13.533Z
   [junit4]   2> 509206 INFO  (zkConnectionManagerCallback-451-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 509222 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 509222 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/control-001/solr.xml
   [junit4]   2> 509225 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 509225 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 509238 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@9eb0fe2, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 509337 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44410/solr
   [junit4]   2> 509367 INFO  (zkConnectionManagerCallback-455-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 509369 INFO  (zkConnectionManagerCallback-457-thread-1-processing-n:127.0.0.1:40559_cg_igk%2Fti) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 509844 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 509844 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:40559_cg_igk%2Fti
   [junit4]   2> 509845 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.Overseer Overseer (id=99341902315651077-127.0.0.1:40559_cg_igk%2Fti-n_0000000000) starting
   [junit4]   2> 509881 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40559_cg_igk%2Fti
   [junit4]   2> 509907 INFO  (zkCallback-456-thread-1-processing-n:127.0.0.1:40559_cg_igk%2Fti) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 510092 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9eb0fe2
   [junit4]   2> 510127 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9eb0fe2
   [junit4]   2> 510127 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9eb0fe2
   [junit4]   2> 510128 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/control-001/cores
   [junit4]   2> 510223 INFO  (zkConnectionManagerCallback-463-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 510224 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 510225 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44410/solr ready
   [junit4]   2> 510831 INFO  (qtp33948457-2459) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:40559_cg_igk%252Fti&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 510911 INFO  (OverseerThreadFactory-944-thread-1-processing-n:127.0.0.1:40559_cg_igk%2Fti) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 511427 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 511427 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 511702 INFO  (zkCallback-456-thread-1-processing-n:127.0.0.1:40559_cg_igk%2Fti) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 512654 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.2.1
   [junit4]   2> 512689 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 513176 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 513216 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 513229 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9eb0fe2
   [junit4]   2> 513229 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 513229 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 513249 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=29, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=30.4716796875, floorSegmentMB=0.8251953125, forceMergeDeletesPctAllowed=29.154419316476925, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5431426969935924
   [junit4]   2> 513251 WARN  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 513507 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 513507 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 513520 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 513520 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 513521 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=0.998046875, floorSegmentMB=0.865234375, forceMergeDeletesPctAllowed=5.790417370401325, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 513521 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@19869d32[control_collection_shard1_replica_n1] main]
   [junit4]   2> 513561 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 513561 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 513562 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 513563 INFO  (searcherExecutor-947-thread-1-processing-n:127.0.0.1:40559_cg_igk%2Fti x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@19869d32[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 513597 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1589470442780884992
   [junit4]   2> 513686 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 513686 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 513686 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:40559/cg_igk/ti/control_collection_shard1_replica_n1/
   [junit4]   2> 513687 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 513687 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:40559/cg_igk/ti/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 513687 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 513724 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:40559/cg_igk/ti/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 513725 INFO  (zkCallback-456-thread-2-processing-n:127.0.0.1:40559_cg_igk%2Fti) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 513780 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 513781 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2354
   [junit4]   2> 513816 INFO  (qtp33948457-2459) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 513906 INFO  (zkCallback-456-thread-2-processing-n:127.0.0.1:40559_cg_igk%2Fti) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 514828 INFO  (qtp33948457-2459) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:40559_cg_igk%252Fti&wt=javabin&version=2} status=0 QTime=3996
   [junit4]   2> 515010 INFO  (OverseerCollectionConfigSetProcessor-99341902315651077-127.0.0.1:40559_cg_igk%2Fti-n_0000000000) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 515034 INFO  (zkConnectionManagerCallback-467-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 515091 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 515092 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44410/solr ready
   [junit4]   2> 515092 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 515113 INFO  (qtp33948457-2460) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 515141 INFO  (OverseerThreadFactory-944-thread-2-processing-n:127.0.0.1:40559_cg_igk%2Fti) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 515141 WARN  (OverseerThreadFactory-944-thread-2-processing-n:127.0.0.1:40559_cg_igk%2Fti) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 515376 INFO  (qtp33948457-2460) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 515376 INFO  (qtp33948457-2460) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=263
   [junit4]   2> 516405 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/shard-1-001 of type NRT
   [junit4]   2> 516406 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 516631 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@9f0a8c2{/cg_igk/ti,null,AVAILABLE}
   [junit4]   2> 516631 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4db4d150{SSL,[ssl, http/1.1]}{127.0.0.1:45879}
   [junit4]   2> 516631 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.e.j.s.Server Started @526351ms
   [junit4]   2> 516631 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/cg_igk/ti, hostPort=45879, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/shard-1-001/cores}
   [junit4]   2> 516632 ERROR (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 516632 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.2.1
   [junit4]   2> 516632 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 516632 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 516632 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-13T09:52:21.002Z
   [junit4]   2> 516687 INFO  (zkConnectionManagerCallback-469-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 516688 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 516688 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/shard-1-001/solr.xml
   [junit4]   2> 516724 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 516724 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 516725 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@9eb0fe2, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 516727 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44410/solr
   [junit4]   2> 516902 INFO  (zkConnectionManagerCallback-473-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 517135 INFO  (zkConnectionManagerCallback-475-thread-1-processing-n:127.0.0.1:45879_cg_igk%2Fti) [n:127.0.0.1:45879_cg_igk%2Fti    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 517140 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:45879_cg_igk%2Fti    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 517157 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:45879_cg_igk%2Fti    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 517159 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:45879_cg_igk%2Fti    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45879_cg_igk%2Fti
   [junit4]   2> 517173 INFO  (zkCallback-466-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 517205 INFO  (zkCallback-456-thread-1-processing-n:127.0.0.1:40559_cg_igk%2Fti) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 517226 INFO  (OverseerCollectionConfigSetProcessor-99341902315651077-127.0.0.1:40559_cg_igk%2Fti-n_0000000000) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 517307 INFO  (zkCallback-474-thread-1-processing-n:127.0.0.1:45879_cg_igk%2Fti) [n:127.0.0.1:45879_cg_igk%2Fti    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 517508 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:45879_cg_igk%2Fti    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9eb0fe2
   [junit4]   2> 517545 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:45879_cg_igk%2Fti    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9eb0fe2
   [junit4]   2> 517546 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:45879_cg_igk%2Fti    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9eb0fe2
   [junit4]   2> 517563 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:45879_cg_igk%2Fti    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/shard-1-001/cores
   [junit4]   2> 517727 INFO  (qtp33948457-2465) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:45879_cg_igk%252Fti&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 517802 INFO  (OverseerThreadFactory-944-thread-3-processing-n:127.0.0.1:40559_cg_igk%2Fti) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:45879_cg_igk%2Fti for creating new replica
   [junit4]   2> 518480 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 518480 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 519556 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.2.1
   [junit4]   2> 519583 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema [collection1_shard2_replica_n41] Schema name=test
   [junit4]   2> 519717 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 519736 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n41' using configuration from collection collection1, trusted=true
   [junit4]   2> 519736 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n41' (registry 'solr.core.collection1.shard2.replica_n41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9eb0fe2
   [junit4]   2> 519737 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 519737 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [[collection1_shard2_replica_n41] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/shard-1-001/cores/collection1_shard2_replica_n41], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/shard-1-001/cores/collection1_shard2_replica_n41/data/]
   [junit4]   2> 519747 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=29, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=30.4716796875, floorSegmentMB=0.8251953125, forceMergeDeletesPctAllowed=29.154419316476925, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5431426969935924
   [junit4]   2> 519749 WARN  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 519790 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 519790 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 519790 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 519791 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 519792 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=0.998046875, floorSegmentMB=0.865234375, forceMergeDeletesPctAllowed=5.790417370401325, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 519792 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.SolrIndexSearcher Opening [Searcher@4c7d15f5[collection1_shard2_replica_n41] main]
   [junit4]   2> 519793 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 519793 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 519793 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 519795 INFO  (searcherExecutor-958-thread-1-processing-n:127.0.0.1:45879_cg_igk%2Fti x:collection1_shard2_replica_n41 s:shard2 c:collection1) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [collection1_shard2_replica_n41] Registered new searcher Searcher@4c7d15f5[collection1_shard2_replica_n41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 519795 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1589470449279959040
   [junit4]   2> 519807 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 519807 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 519807 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:45879/cg_igk/ti/collection1_shard2_replica_n41/
   [junit4]   2> 519807 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 519807 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy https://127.0.0.1:45879/cg_igk/ti/collection1_shard2_replica_n41/ has no replicas
   [junit4]   2> 519807 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 519809 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:45879/cg_igk/ti/collection1_shard2_replica_n41/ shard2
   [junit4]   2> 519960 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 519961 INFO  (qtp1502012540-2513) [n:127.0.0.1:45879_cg_igk%2Fti c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1481
   [junit4]   2> 519963 INFO  (qtp33948457-2465) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:45879_cg_igk%252Fti&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=2235
   [junit4]   2> 520080 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/shard-2-001 of type NRT
   [junit4]   2> 520081 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 520082 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@24f2d994{/cg_igk/ti,null,AVAILABLE}
   [junit4]   2> 520082 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3727cdbd{SSL,[ssl, http/1.1]}{127.0.0.1:44093}
   [junit4]   2> 520082 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.e.j.s.Server Started @529802ms
   [junit4]   2> 520083 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/cg_igk/ti, hostPort=44093, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/shard-2-001/cores}
   [junit4]   2> 520083 ERROR (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 520083 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.2.1
   [junit4]   2> 520083 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 520083 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 520083 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-13T09:52:24.453Z
   [junit4]   2> 520085 INFO  (zkConnectionManagerCallback-478-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 520086 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 520086 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/shard-2-001/solr.xml
   [junit4]   2> 520089 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 520089 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 520098 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@9eb0fe2, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 520101 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44410/solr
   [junit4]   2> 520110 INFO  (zkConnectionManagerCallback-482-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 520112 INFO  (zkConnectionManagerCallback-484-thread-1-processing-n:127.0.0.1:44093_cg_igk%2Fti) [n:127.0.0.1:44093_cg_igk%2Fti    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 520116 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:44093_cg_igk%2Fti    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 520117 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:44093_cg_igk%2Fti    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 520118 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:44093_cg_igk%2Fti    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44093_cg_igk%2Fti
   [junit4]   2> 520119 INFO  (zkCallback-466-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 520119 INFO  (zkCallback-474-thread-1-processing-n:127.0.0.1:45879_cg_igk%2Fti) [n:127.0.0.1:45879_cg_igk%2Fti    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 520119 INFO  (zkCallback-456-thread-2-processing-n:127.0.0.1:40559_cg_igk%2Fti) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 520154 INFO  (zkCallback-483-thread-1-processing-n:127.0.0.1:44093_cg_igk%2Fti) [n:127.0.0.1:44093_cg_igk%2Fti    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 520241 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:44093_cg_igk%2Fti    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9eb0fe2
   [junit4]   2> 520257 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:44093_cg_igk%2Fti    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9eb0fe2
   [junit4]   2> 520257 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:44093_cg_igk%2Fti    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9eb0fe2
   [junit4]   2> 520258 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [n:127.0.0.1:44093_cg_igk%2Fti    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/shard-2-001/cores
   [junit4]   2> 520284 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:44093_cg_igk%252Fti&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 520286 INFO  (OverseerCollectionConfigSetProcessor-99341902315651077-127.0.0.1:40559_cg_igk%2Fti-n_0000000000) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 520286 INFO  (OverseerThreadFactory-944-thread-4-processing-n:127.0.0.1:40559_cg_igk%2Fti) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:44093_cg_igk%2Fti for creating new replica
   [junit4]   2> 520343 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 520343 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 521355 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.2.1
   [junit4]   2> 521385 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.IndexSchema [collection1_shard1_replica_n43] Schema name=test
   [junit4]   2> 521521 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 521552 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n43' using configuration from collection collection1, trusted=true
   [junit4]   2> 521552 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n43' (registry 'solr.core.collection1.shard1.replica_n43') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9eb0fe2
   [junit4]   2> 521552 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 521552 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrCore [[collection1_shard1_replica_n43] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/shard-2-001/cores/collection1_shard1_replica_n43], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/shard-2-001/cores/collection1_shard1_replica_n43/data/]
   [junit4]   2> 521555 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=29, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=30.4716796875, floorSegmentMB=0.8251953125, forceMergeDeletesPctAllowed=29.154419316476925, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5431426969935924
   [junit4]   2> 521558 WARN  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 521631 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 521631 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 521632 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 521632 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 521641 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=0.998046875, floorSegmentMB=0.865234375, forceMergeDeletesPctAllowed=5.790417370401325, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 521641 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.SolrIndexSearcher Opening [Searcher@49d1476b[collection1_shard1_replica_n43] main]
   [junit4]   2> 521642 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 521642 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 521643 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 521666 INFO  (searcherExecutor-969-thread-1-processing-n:127.0.0.1:44093_cg_igk%2Fti x:collection1_shard1_replica_n43 s:shard1 c:collection1) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrCore [collection1_shard1_replica_n43] Registered new searcher Searcher@49d1476b[collection1_shard1_replica_n43] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 521666 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1589470451241844736
   [junit4]   2> 521683 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 521683 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 521683 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:44093/cg_igk/ti/collection1_shard1_replica_n43/
   [junit4]   2> 521683 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 521683 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy https://127.0.0.1:44093/cg_igk/ti/collection1_shard1_replica_n43/ has no replicas
   [junit4]   2> 521683 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 521693 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:44093/cg_igk/ti/collection1_shard1_replica_n43/ shard1
   [junit4]   2> 521844 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 521845 INFO  (qtp2005904717-2544) [n:127.0.0.1:44093_cg_igk%2Fti c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1502
   [junit4]   2> 521847 INFO  (qtp33948457-2466) [n:127.0.0.1:40559_cg_igk%2Fti    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:44093_cg_igk%252Fti&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1563
   [junit4]   2> 522009 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/shard-3-001 of type NRT
   [junit4]   2> 522010 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 522011 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1b20c166{/cg_igk/ti,null,AVAILABLE}
   [junit4]   2> 522012 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6e70e857{SSL,[ssl, http/1.1]}{127.0.0.1:40929}
   [junit4]   2> 522012 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.e.j.s.Server Started @531731ms
   [junit4]   2> 522012 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/cg_igk/ti, hostPort=40929, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/shard-3-001/cores}
   [junit4]   2> 522012 ERROR (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 522013 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.2.1
   [junit4]   2> 522013 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 522013 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 522013 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-13T09:52:26.383Z
   [junit4]   2> 522014 INFO  (zkConnectionManagerCallback-487-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 522015 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 522015 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_5F2751DD4FCACFEF-001/shard-3-001/solr.xml
   [junit4]   2> 522019 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 522019 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 522020 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@9eb0fe2, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 522022 INFO  (TEST-MultiThreadedOCPTest.test-seed#[5F2751DD4FCACFEF]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44410/solr
   [junit4]   2> 522023 INFO  (zkConnectionManagerCallback-491-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 522025 INFO  (zkConnectionManagerCallback-493-thread-1-processing-n:127.0.0.1:40929_cg_igk%2Fti) [n:127.0.0.1:40929_cg_igk%2Fti    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 5

[...truncated too long message...]

able #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 6 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:
   [junit4]    >    1) Thread[id=23176, name=zkConnectionManagerCallback-3721-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    2) Thread[id=23175, name=StoppableIndexingThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   [junit4]    >    3) Thread[id=23174, name=StoppableIndexingThread-SendThread(127.0.0.1:41803), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   [junit4]    >    4) Thread[id=23265, name=zkCallback-3720-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    5) Thread[id=23177, name=zkCallback-3720-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    6) Thread[id=23266, name=zkCallback-3720-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([5F2751DD4FCACFEF]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=23176, name=zkConnectionManagerCallback-3721-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    2) Thread[id=23174, name=StoppableIndexingThread-SendThread(127.0.0.1:41803), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   [junit4]    >    3) Thread[id=23265, name=zkCallback-3720-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    4) Thread[id=23177, name=zkCallback-3720-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    5) Thread[id=23266, name=zkCallback-3720-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([5F2751DD4FCACFEF]:0)
   [junit4] Completed [317/751 (2!)] on J2 in 116.12s, 1 test, 1 failure, 2 errors <<< FAILURES!

[...truncated 48628 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.2 - Build # 9 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.2/9/

2 tests failed.
FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication

Error Message:
timed out waiting for collection1 startAt time to exceed: Sun Jan 14 19:06:55 AST 2018

Stack Trace:
java.lang.AssertionError: timed out waiting for collection1 startAt time to exceed: Sun Jan 14 19:06:55 AST 2018
        at __randomizedtesting.SeedInfo.seed([B98C80A4FD974A51:4EFF6EFC3B7FE5B7]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.handler.TestReplicationHandler.watchCoreStartAt(TestReplicationHandler.java:1565)
        at org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication(TestReplicationHandler.java:1364)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:
Expected numSlices=5 numReplicas=1 but found DocCollection(solrj_collection3//collections/solrj_collection3/state.json/27)={   "pullReplicas":"0",   "replicationFactor":"1",   "shards":{     "shard1":{       "range":"80000000-b332ffff",       "state":"active",       "replicas":{"core_node3":{           "dataDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node3/data/",           "base_url":"http://127.0.0.1:43724/f_gsj",           "node_name":"127.0.0.1:43724_f_gsj",           "type":"NRT",           "ulogDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node3/data/tlog",           "core":"solrj_collection3_shard1_replica_n1",           "shared_storage":"true",           "state":"active",           "leader":"true"}}},     "shard2":{       "range":"b3330000-e665ffff",       "state":"active",       "replicas":{"core_node5":{           "dataDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node5/data/",           "base_url":"http://127.0.0.1:36886/f_gsj",           "node_name":"127.0.0.1:36886_f_gsj",           "type":"NRT",           "ulogDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node5/data/tlog",           "core":"solrj_collection3_shard2_replica_n2",           "shared_storage":"true",           "state":"active",           "leader":"true"}}},     "shard3":{       "range":"e6660000-1998ffff",       "state":"active",       "replicas":{"core_node7":{           "dataDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node7/data/",           "base_url":"http://127.0.0.1:41232/f_gsj",           "node_name":"127.0.0.1:41232_f_gsj",           "type":"NRT",           "ulogDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node7/data/tlog",           "core":"solrj_collection3_shard3_replica_n4",           "shared_storage":"true",           "state":"active",           "leader":"true"}}},     "shard4":{       "range":"19990000-4ccbffff",       "state":"active",       "replicas":{"core_node9":{           "dataDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node9/data/",           "base_url":"http://127.0.0.1:36886/f_gsj",           "node_name":"127.0.0.1:36886_f_gsj",           "type":"NRT",           "ulogDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node9/data/tlog",           "core":"solrj_collection3_shard4_replica_n6",           "shared_storage":"true",           "state":"active",           "leader":"true"}}},     "shard5":{       "range":"4ccc0000-7fffffff",       "state":"active",       "replicas":{"core_node10":{           "core":"solrj_collection3_shard5_replica_n8",           "base_url":"http://127.0.0.1:41232/f_gsj",           "node_name":"127.0.0.1:41232_f_gsj",           "state":"down",           "type":"NRT"}}}},   "router":{     "field":"myOwnField",     "name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"true",   "nrtReplicas":"1",   "tlogReplicas":"0"} with /live_nodes: [127.0.0.1:33811_f_gsj, 127.0.0.1:36886_f_gsj, 127.0.0.1:41232_f_gsj, 127.0.0.1:43724_f_gsj, 127.0.0.1:44683_f_gsj]

Stack Trace:
java.lang.AssertionError: Expected numSlices=5 numReplicas=1 but found DocCollection(solrj_collection3//collections/solrj_collection3/state.json/27)={
  "pullReplicas":"0",
  "replicationFactor":"1",
  "shards":{
    "shard1":{
      "range":"80000000-b332ffff",
      "state":"active",
      "replicas":{"core_node3":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node3/data/",
          "base_url":"http://127.0.0.1:43724/f_gsj",
          "node_name":"127.0.0.1:43724_f_gsj",
          "type":"NRT",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node3/data/tlog",
          "core":"solrj_collection3_shard1_replica_n1",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}},
    "shard2":{
      "range":"b3330000-e665ffff",
      "state":"active",
      "replicas":{"core_node5":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node5/data/",
          "base_url":"http://127.0.0.1:36886/f_gsj",
          "node_name":"127.0.0.1:36886_f_gsj",
          "type":"NRT",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node5/data/tlog",
          "core":"solrj_collection3_shard2_replica_n2",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}},
    "shard3":{
      "range":"e6660000-1998ffff",
      "state":"active",
      "replicas":{"core_node7":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node7/data/",
          "base_url":"http://127.0.0.1:41232/f_gsj",
          "node_name":"127.0.0.1:41232_f_gsj",
          "type":"NRT",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node7/data/tlog",
          "core":"solrj_collection3_shard3_replica_n4",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}},
    "shard4":{
      "range":"19990000-4ccbffff",
      "state":"active",
      "replicas":{"core_node9":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node9/data/",
          "base_url":"http://127.0.0.1:36886/f_gsj",
          "node_name":"127.0.0.1:36886_f_gsj",
          "type":"NRT",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node9/data/tlog",
          "core":"solrj_collection3_shard4_replica_n6",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}},
    "shard5":{
      "range":"4ccc0000-7fffffff",
      "state":"active",
      "replicas":{"core_node10":{
          "core":"solrj_collection3_shard5_replica_n8",
          "base_url":"http://127.0.0.1:41232/f_gsj",
          "node_name":"127.0.0.1:41232_f_gsj",
          "state":"down",
          "type":"NRT"}}}},
  "router":{
    "field":"myOwnField",
    "name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"true",
  "nrtReplicas":"1",
  "tlogReplicas":"0"} with /live_nodes: [127.0.0.1:33811_f_gsj, 127.0.0.1:36886_f_gsj, 127.0.0.1:41232_f_gsj, 127.0.0.1:43724_f_gsj, 127.0.0.1:44683_f_gsj]
        at __randomizedtesting.SeedInfo.seed([B98C80A4FD974A51:31D8BF7E536B27A9]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.assertSliceAndReplicaCount(SharedFSAutoReplicaFailoverTest.java:413)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:255)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:143)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13412 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/init-core-data-001
   [junit4]   2> 3012481 WARN  (SUITE-TestReplicationHandler-seed#[B98C80A4FD974A51]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=5 numCloses=5
   [junit4]   2> 3012481 INFO  (SUITE-TestReplicationHandler-seed#[B98C80A4FD974A51]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 3012482 INFO  (SUITE-TestReplicationHandler-seed#[B98C80A4FD974A51]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 3012482 INFO  (SUITE-TestReplicationHandler-seed#[B98C80A4FD974A51]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 3012483 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestReplicateAfterStartup
   [junit4]   2> 3012484 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-001/collection1
   [junit4]   2> 3012486 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 3012529 ERROR (Finalizer) [    ] o.a.s.c.CoreContainer CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=1840689974
   [junit4]   2> 3012529 ERROR (Finalizer) [    ] o.a.s.c.CoreContainer CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=1189756551
   [junit4]   2> 3012529 ERROR (Finalizer) [    ] o.a.s.c.CoreContainer CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=2121241459
   [junit4]   2> 3012529 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4117ebd2{/solr,null,AVAILABLE}
   [junit4]   2> 3012530 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2de76501{HTTP/1.1,[http/1.1]}{127.0.0.1:38569}
   [junit4]   2> 3012530 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.e.j.s.Server Started @3023358ms
   [junit4]   2> 3012530 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=38569}
   [junit4]   2> 3012531 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3012531 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.2.1
   [junit4]   2> 3012531 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 3012531 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3012531 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-14T16:04:23.256Z
   [junit4]   2> 3012531 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-001/solr.xml
   [junit4]   2> 3012542 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 3012542 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 3012543 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@ee3f827, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3012581 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ee3f827
   [junit4]   2> 3012589 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ee3f827
   [junit4]   2> 3012589 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ee3f827
   [junit4]   2> 3012591 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-001/.
   [junit4]   2> 3012591 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3012591 INFO  (coreLoadExecutor-4745-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3012599 INFO  (coreLoadExecutor-4745-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.2.1
   [junit4]   2> 3012614 INFO  (coreLoadExecutor-4745-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3012616 INFO  (coreLoadExecutor-4745-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 3012616 INFO  (coreLoadExecutor-4745-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 3012616 INFO  (coreLoadExecutor-4745-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ee3f827
   [junit4]   2> 3012616 INFO  (coreLoadExecutor-4745-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3012616 INFO  (coreLoadExecutor-4745-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-001/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 3012629 INFO  (coreLoadExecutor-4745-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=42, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3012658 INFO  (coreLoadExecutor-4745-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3012658 INFO  (coreLoadExecutor-4745-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3012659 INFO  (coreLoadExecutor-4745-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=49, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3012659 INFO  (coreLoadExecutor-4745-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3e6ba7ed[collection1] main]
   [junit4]   2> 3012659 INFO  (coreLoadExecutor-4745-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-001/collection1/conf
   [junit4]   2> 3012660 INFO  (coreLoadExecutor-4745-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 3012660 INFO  (coreLoadExecutor-4745-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3012660 INFO  (searcherExecutor-4746-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3e6ba7ed[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3012661 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-002/collection1
   [junit4]   2> 3012663 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 3012685 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@27aef666{/solr,null,AVAILABLE}
   [junit4]   2> 3012686 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@71a757bd{HTTP/1.1,[http/1.1]}{127.0.0.1:37909}
   [junit4]   2> 3012686 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.e.j.s.Server Started @3023514ms
   [junit4]   2> 3012686 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=37909}
   [junit4]   2> 3012686 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3012686 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.2.1
   [junit4]   2> 3012687 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 3012687 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3012687 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-14T16:04:23.412Z
   [junit4]   2> 3012687 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-002/solr.xml
   [junit4]   2> 3012690 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 3012690 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 3012691 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@ee3f827, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3012730 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ee3f827
   [junit4]   2> 3012739 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ee3f827
   [junit4]   2> 3012739 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ee3f827
   [junit4]   2> 3012741 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-002/.
   [junit4]   2> 3012741 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3012741 INFO  (coreLoadExecutor-4755-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3012770 INFO  (coreLoadExecutor-4755-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.2.1
   [junit4]   2> 3012800 INFO  (coreLoadExecutor-4755-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3012802 INFO  (coreLoadExecutor-4755-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 3012802 INFO  (coreLoadExecutor-4755-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 3012802 INFO  (coreLoadExecutor-4755-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ee3f827
   [junit4]   2> 3012802 INFO  (coreLoadExecutor-4755-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3012802 INFO  (coreLoadExecutor-4755-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-002/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 3012804 INFO  (coreLoadExecutor-4755-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=42, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3012905 INFO  (coreLoadExecutor-4755-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3012905 INFO  (coreLoadExecutor-4755-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3012906 INFO  (coreLoadExecutor-4755-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=49, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3012906 INFO  (coreLoadExecutor-4755-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4474a23[collection1] main]
   [junit4]   2> 3012907 INFO  (coreLoadExecutor-4755-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-002/collection1/conf
   [junit4]   2> 3012908 INFO  (coreLoadExecutor-4755-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 3012908 INFO  (coreLoadExecutor-4755-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3012909 INFO  (searcherExecutor-4756-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4474a23[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3012909 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@71a757bd{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 3012909 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=123214784
   [junit4]   2> 3012909 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 3012910 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7d981c70: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@6d8cec27
   [junit4]   2> 3012915 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 3012915 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1ac51625: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@7a25c98a
   [junit4]   2> 3012935 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 3012935 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@73758b12: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@717c7fc3
   [junit4]   2> 3012935 INFO  (coreCloseExecutor-4761-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@e361427
   [junit4]   2> 3012935 INFO  (coreCloseExecutor-4761-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=238425127
   [junit4]   2> 3012935 INFO  (coreCloseExecutor-4761-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@289e4be4: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@13e903f1
   [junit4]   2> 3012946 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@27aef666{/solr,null,UNAVAILABLE}
   [junit4]   2> 3012947 INFO  (qtp1299830695-39491) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 3012947 INFO  (qtp1299830695-39491) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 0
   [junit4]   2> 3012947 INFO  (qtp1299830695-39492) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 3012948 INFO  (qtp1299830695-39492) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@197c97c3 commitCommandVersion:0
   [junit4]   2> 3012948 INFO  (qtp1299830695-39492) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@568660a7[collection1] main]
   [junit4]   2> 3012949 INFO  (qtp1299830695-39492) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 3012949 INFO  (searcherExecutor-4746-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@568660a7[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3012949 INFO  (qtp1299830695-39492) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 1
   [junit4]   2> 3012957 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@2de76501{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 3012957 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=493448799
   [junit4]   2> 3012957 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 3012957 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3e80bbf3: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@29f3d905
   [junit4]   2> 3012958 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 3012958 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@38586168: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@7a25c98a
   [junit4]   2> 3012959 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 3012959 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@e7e57f9: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@717c7fc3
   [junit4]   2> 3012959 INFO  (coreCloseExecutor-4763-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3c9eaf13
   [junit4]   2> 3012959 INFO  (coreCloseExecutor-4763-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1017032467
   [junit4]   2> 3012960 INFO  (coreCloseExecutor-4763-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6d020051: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@7b8eed79
   [junit4]   2> 3012962 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4117ebd2{/solr,null,UNAVAILABLE}
   [junit4]   2> 3012963 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 3012963 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@734d88ef{/solr,null,AVAILABLE}
   [junit4]   2> 3012963 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@bf02a5{HTTP/1.1,[http/1.1]}{127.0.0.1:40477}
   [junit4]   2> 3012964 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.e.j.s.Server Started @3023792ms
   [junit4]   2> 3012964 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=40477}
   [junit4]   2> 3012964 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3012964 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.2.1
   [junit4]   2> 3012964 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 3012964 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3012964 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-14T16:04:23.689Z
   [junit4]   2> 3012965 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-001/solr.xml
   [junit4]   2> 3012967 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 3012967 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 3012981 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@ee3f827, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3013031 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ee3f827
   [junit4]   2> 3013063 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ee3f827
   [junit4]   2> 3013063 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ee3f827
   [junit4]   2> 3013064 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-001/.
   [junit4]   2> 3013064 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[B98C80A4FD974A51]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3013065 INFO  (coreLoadExecutor-4770-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3013119 INFO  (coreLoadExecutor-4770-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.2.1
   [junit4]   2> 3013138 INFO  (coreLoadExecutor-4770-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3013152 INFO  (coreLoadExecutor-4770-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 3013152 INFO  (coreLoadExecutor-4770-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 3013152 INFO  (coreLoadExecutor-4770-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ee3f827
   [junit4]   2> 3013152 INFO  (coreLoadExecutor-4770-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3013153 INFO  (coreLoadExecutor-4770-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-001/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 3013154 INFO  (coreLoadExecutor-4770-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=42, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3013298 INFO  (coreLoadExecutor-4770-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3013298 INFO  (coreLoadExecutor-4770-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3013299 INFO  (coreLoadExecutor-4770-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=49, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3013299 INFO  (coreLoadExecutor-4770-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6e5debcb[collection1] main]
   [junit4]   2> 3013299 INFO  (coreLoadExecutor-4770-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_B98C80A4FD974A51-001/solr-instance-001/collection1/conf
   [junit4]   2> 3013299 INFO  (coreLoadExecutor-4770-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml
   [junit4]   2> 3013299 INFO  (coreLoadExecutor-4770-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3013300 INFO  (searcherExecutor-4771-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6e5debcb[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3013326 INFO  (qtp1549177328-39533) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 19
   [junit4]   2> 3013326 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 0
   [junit4]   2> 3013327 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2]} 0 0
   [junit4]   2> 3013327 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[3]} 0 0
   [junit4]   2> 3013327 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[4]} 0 0
   [junit4]   2> 3013327 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[5]} 0 0
   [junit4]   2> 3013328 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[6]} 0 0
   [junit4]   2> 3013328 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[7]} 0 0
   [junit4]   2> 3013328 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[8]} 0 0
   [junit4]   2> 3013328 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9]} 0 0
   [junit4]   2> 3013328 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[10]} 0 0
   [junit4]   2> 3013329 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[11]} 0 0
   [junit4]   2> 3013341 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[12]} 0 0
   [junit4]   2> 3013341 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[13]} 0 0
   [junit4]   2> 3013342 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[14]} 0 0
   [junit4]   2> 3013342 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[15]} 0 0
   [junit4]   2> 3013342 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[16]} 0 0
   [junit4]   2> 3013342 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[17]} 0 0
   [junit4]   2> 3013342 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[18]} 0 0
   [junit4]   2> 3013343 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[19]} 0 0
   [junit4]   2> 3013343 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[20]} 0 0
   [junit4]   2> 3013343 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[21]} 0 0
   [junit4]   2> 3013343 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[22]} 0 0
   [junit4]   2> 3013344 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[23]} 0 0
   [junit4]   2> 3013344 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[24]} 0 0
   [junit4]   2> 3013344 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[25]} 0 0
   [junit4]   2> 3013344 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[26]} 0 0
   [junit4]   2> 3013344 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[27]} 0 0
   [junit4]   2> 3013345 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[28]} 0 0
   [junit4]   2> 3013345 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[29]} 0 0
   [junit4]   2> 3013345 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[30]} 0 0
   [junit4]   2> 3013361 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[31]} 0 0
   [junit4]   2> 3013362 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[32]} 0 0
   [junit4]   2> 3013362 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[33]} 0 0
   [junit4]   2> 3013362 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[34]} 0 0
   [junit4]   2> 3013363 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[35]} 0 0
   [junit4]   2> 3013363 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[36]} 0 0
   [junit4]   2> 3013363 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[37]} 0 0
   [junit4]   2> 3013363 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[38]} 0 0
   [junit4]   2> 3013364 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[39]} 0 0
   [junit4]   2> 3013364 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[40]} 0 0
   [junit4]   2> 3013364 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[41]} 0 0
   [junit4]   2> 3013364 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[42]} 0 0
   [junit4]   2> 3013365 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[43]} 0 0
   [junit4]   2> 3013365 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[44]} 0 0
   [junit4]   2> 3013365 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[45]} 0 0
   [junit4]   2> 3013365 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[46]} 0 0
   [junit4]   2> 3013378 INFO  (qtp1549177328-39534) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[47]} 0 0
   [junit4]   2> 3013378 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[48]} 0 0
   [junit4]   2> 3013378 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[49]} 0 0
   [junit4]   2> 3013378 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[50]} 0 0
   [junit4]   2> 3013379 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[51]} 0 0
   [junit4]   2> 3013379 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[52]} 0 0
   [junit4]   2> 3013379 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[53]} 0 0
   [junit4]   2> 3013379 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[54]} 0 0
   [junit4]   2> 3013380 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[55]} 0 0
   [junit4]   2> 3013380 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[56]} 0 0
   [junit4]   2> 3013380 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[57]} 0 0
   [junit4]   2> 3013380 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[58]} 0 0
   [junit4]   2> 3013381 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[59]} 0 0
   [junit4]   2> 3013381 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[60]} 0 0
   [junit4]   2> 3013381 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[61]} 0 0
   [junit4]   2> 3013381 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[62]} 0 0
   [junit4]   2> 3013381 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[63]} 0 0
   [junit4]   2> 3013398 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[64]} 0 0
   [junit4]   2> 3013398 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[65]} 0 0
   [junit4]   2> 3013398 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[66]} 0 0
   [junit4]   2> 3013399 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[67]} 0 0
   [junit4]   2> 3013399 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[68]} 0 0
   [junit4]   2> 3013399 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[69]} 0 0
   [junit4]   2> 3013399 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[70]} 0 0
   [junit4]   2> 3013400 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[71]} 0 0
   [junit4]   2> 3013400 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[72]} 0 0
   [junit4]   2> 3013400 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[73]} 0 0
   [junit4]   2> 3013400 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[74]} 0 0
   [junit4]   2> 3013400 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[75]} 0 0
   [junit4]   2> 3013401 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[76]} 0 0
   [junit4]   2> 3013401 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[77]} 0 0
   [junit4]   2> 3013401 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[78]} 0 0
   [junit4]   2> 3013401 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[79]} 0 0
   [junit4]   2> 3013402 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[80]} 0 0
   [junit4]   2> 3013414 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[81]} 0 12
   [junit4]   2> 3013414 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[82]} 0 0
   [junit4]   2> 3013415 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[83]} 0 0
   [junit4]   2> 3013415 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[84]} 0 0
   [junit4]   2> 3013415 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[85]} 0 0
   [junit4]   2> 3013415 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[86]} 0 0
   [junit4]   2> 3013415 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[87]} 0 0
   [junit4]   2> 3013416 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[88]} 0 0
   [junit4]   2> 3013416 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[89]} 0 0
   [junit4]   2> 3013416 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[90]} 0 0
   [junit4]   2> 3013416 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[91]} 0 0
   [junit4]   2> 3013416 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[92]} 0 0
   [junit4]   2> 3013417 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[93]} 0 0
   [junit4]   2> 3013417 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[94]} 0 0
   [junit4]   2> 3013417 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[95]} 0 0
   [junit4]   2> 3013417 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[96]} 0 0
   [junit4]   2> 3013417 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[97]} 0 0
   [junit4]   2> 3013418 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[98]} 0 0
   [junit4]   2> 3013418 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[99]} 0 0
   [junit4]   2> 3013451 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[100]} 0 32
   [junit4]   2> 3013451 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[101]} 0 0
   [junit4]   2> 3013451 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[102]} 0 0
   [junit4]   2> 3013452 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[103]} 0 0
   [junit4]   2> 3013452 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[104]} 0 0
   [junit4]   2> 3013452 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[105]} 0 0
   [junit4]   2> 3013452 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[106]} 0 0
   [junit4]   2> 3013452 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[107]} 0 0
   [junit4]   2> 3013453 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[108]} 0 0
   [junit4]   2> 3013453 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[109]} 0 0
   [junit4]   2> 3013453 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[110]} 0 0
   [junit4]   2> 3013453 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[111]} 0 0
   [junit4]   2> 3013453 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[112]} 0 0
   [junit4]   2> 3013454 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[113]} 0 0
   [junit4]   2> 3013454 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[114]} 0 0
   [junit4]   2> 3013454 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[115]} 0 0
   [junit4]   2> 3013454 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[116]} 0 0
   [junit4]   2> 3013471 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[117]} 0 16
   [junit4]   2> 3013471 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[118]} 0 0
   [junit4]   2> 3013472 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[119]} 0 0
   [junit4]   2> 3013472 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[120]} 0 0
   [junit4]   2> 3013472 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[121]} 0 0
   [junit4]   2> 3013472 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[122]} 0 0
   [junit4]   2> 3013472 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[123]} 0 0
   [junit4]   2> 3013473 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[124]} 0 0
   [junit4]   2> 3013473 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[125]} 0 0
   [junit4]   2> 3013473 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[126]} 0 0
   [junit4]   2> 3013473 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[127]} 0 0
   [junit4]   2> 3013474 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[128]} 0 0
   [junit4]   2> 3013474 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[129]} 0 0
   [junit4]   2> 3013474 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[130]} 0 0
   [junit4]   2> 3013474 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[131]} 0 0
   [junit4]   2> 3013474 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[132]} 0 0
   [junit4]   2> 3013475 INFO  (qtp1549177328-39536) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[133]} 0 0
   [junit4]   2> 3013487 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[134]} 0 0
   [junit4]   2> 3013508 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[135]} 0 0
   [junit4]   2> 3013508 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[136]} 0 0
   [junit4]   2> 3013508 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[137]} 0 0
   [junit4]   2> 3013508 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[138]} 0 0
   [junit4]   2> 3013509 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[139]} 0 0
   [junit4]   2> 3013509 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[140]} 0 0
   [junit4]   2> 3013509 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[141]} 0 0
   [junit4]   2> 3013509 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[142]} 0 0
   [junit4]   2> 3013510 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[143]} 0 0
   [junit4]   2> 3013510 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[144]} 0 0
   [junit4]   2> 3013510 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[145]} 0 0
   [junit4]   2> 3013510 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[146]} 0 0
   [junit4]   2> 3013511 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[147]} 0 0
   [junit4]   2> 3013511 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[148]} 0 0
   [junit4]   2> 3013511 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[149]} 0 0
   [junit4]   2> 3013523 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[150]} 0 0
   [junit4]   2> 3013524 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[151]} 0 0
   [junit4]   2> 3013524 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[152]} 0 0
   [junit4]   2> 3013524 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[153]} 0 0
   [junit4]   2> 3013525 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[154]} 0 0
   [junit4]   2> 3013525 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[155]} 0 0
   [junit4]   2> 3013525 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[156]} 0 0
   [junit4]   2> 3013525 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[157]} 0 0
   [junit4]   2> 3013525 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[158]} 0 0
   [junit4]   2> 3013526 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[159]} 0 0
   [junit4]   2> 3013526 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[160]} 0 0
   [junit4]   2> 3013526 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[161]} 0 0
   [junit4]   2> 3013526 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[162]} 0 0
   [junit4]   2> 3013527 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[163]} 0 0
   [junit4]   2> 3013527 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[164]} 0 0
   [junit4]   2> 3013527 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[165]} 0 0
   [junit4]   2> 3013540 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[166]} 0 12
   [junit4]   2> 3013540 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[167]} 0 0
   [junit4]   2> 3013540 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[168]} 0 0
   [junit4]   2> 3013540 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[169]} 0 0
   [junit4]   2> 3013541 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[170]} 0 0
   [junit4]   2> 3013541 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[171]} 0 0
   [junit4]   2> 3013541 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[172]} 0 0
   [junit4]   2> 3013541 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[173]} 0 0
   [junit4]   2> 3013541 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[174]} 0 0
   [junit4]   2> 3013542 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[175]} 0 0
   [junit4]   2> 3013542 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[176]} 0 0
   [junit4]   2> 3013542 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[177]} 0 0
   [junit4]   2> 3013542 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[178]} 0 0
   [junit4]   2> 3013542 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[179]} 0 0
   [junit4]   2> 3013543 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[180]} 0 0
   [junit4]   2> 3013543 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[181]} 0 0
   [junit4]   2> 3013543 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[182]} 0 0
   [junit4]   2> 3013543 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[183]} 0 0
   [junit4]   2> 3013543 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[184]} 0 0
   [junit4]   2> 3013544 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[185]} 0 0
   [junit4]   2> 3013580 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[186]} 0 0
   [junit4]   2> 3013581 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[187]} 0 0
   [junit4]   2> 3013581 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[188]} 0 0
   [junit4]   2> 3013581 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[189]} 0 0
   [junit4]   2> 3013581 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[190]} 0 0
   [junit4]   2> 3013582 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[191]} 0 0
   [junit4]   2> 3013582 INFO  (qtp1549177328-39535) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[192]} 0 0
   [junit4]   2> 3013613 INFO  (qtp1549177328-39530) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[193]} 0 0
   [junit4]   2> 3013613 INFO  (qtp1549177328-39533) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&ve

[...truncated too long message...]

j_collection3/core_node7/data/",
   [junit4]    >           "base_url":"http://127.0.0.1:41232/f_gsj",
   [junit4]    >           "node_name":"127.0.0.1:41232_f_gsj",
   [junit4]    >           "type":"NRT",
   [junit4]    >           "ulogDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node7/data/tlog",
   [junit4]    >           "core":"solrj_collection3_shard3_replica_n4",
   [junit4]    >           "shared_storage":"true",
   [junit4]    >           "state":"active",
   [junit4]    >           "leader":"true"}}},
   [junit4]    >     "shard4":{
   [junit4]    >       "range":"19990000-4ccbffff",
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{"core_node9":{
   [junit4]    >           "dataDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node9/data/",
   [junit4]    >           "base_url":"http://127.0.0.1:36886/f_gsj",
   [junit4]    >           "node_name":"127.0.0.1:36886_f_gsj",
   [junit4]    >           "type":"NRT",
   [junit4]    >           "ulogDir":"hdfs://lucene2-us-west.apache.org:34162/solr_hdfs_home/solrj_collection3/core_node9/data/tlog",
   [junit4]    >           "core":"solrj_collection3_shard4_replica_n6",
   [junit4]    >           "shared_storage":"true",
   [junit4]    >           "state":"active",
   [junit4]    >           "leader":"true"}}},
   [junit4]    >     "shard5":{
   [junit4]    >       "range":"4ccc0000-7fffffff",
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{"core_node10":{
   [junit4]    >           "core":"solrj_collection3_shard5_replica_n8",
   [junit4]    >           "base_url":"http://127.0.0.1:41232/f_gsj",
   [junit4]    >           "node_name":"127.0.0.1:41232_f_gsj",
   [junit4]    >           "state":"down",
   [junit4]    >           "type":"NRT"}}}},
   [junit4]    >   "router":{
   [junit4]    >     "field":"myOwnField",
   [junit4]    >     "name":"compositeId"},
   [junit4]    >   "maxShardsPerNode":"1",
   [junit4]    >   "autoAddReplicas":"true",
   [junit4]    >   "nrtReplicas":"1",
   [junit4]    >   "tlogReplicas":"0"} with /live_nodes: [127.0.0.1:33811_f_gsj, 127.0.0.1:36886_f_gsj, 127.0.0.1:41232_f_gsj, 127.0.0.1:43724_f_gsj, 127.0.0.1:44683_f_gsj]
   [junit4]    > at __randomizedtesting.SeedInfo.seed([B98C80A4FD974A51:31D8BF7E536B27A9]:0)
   [junit4]    > at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.assertSliceAndReplicaCount(SharedFSAutoReplicaFailoverTest.java:413)
   [junit4]    > at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:255)
   [junit4]    > at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:143)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 3973171 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[B98C80A4FD974A51]-worker) [    ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 3973216 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[B98C80A4FD974A51]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 3973317 WARN  (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_B98C80A4FD974A51-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_B98C80A4FD974A51-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:34162) [    ] o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager interrupted
   [junit4]   2> 3973317 WARN  (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_B98C80A4FD974A51-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_B98C80A4FD974A51-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:34162) [    ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1739871814-127.0.0.1-1515945419444 (Datanode Uuid 465780e2-874c-4c49-9fa9-10cc24cf13a4) service to lucene2-us-west.apache.org/127.0.0.1:34162
   [junit4]   2> 3973319 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[B98C80A4FD974A51]-worker) [    ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 3973383 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[B98C80A4FD974A51]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 3973484 WARN  (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_B98C80A4FD974A51-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_B98C80A4FD974A51-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:34162) [    ] o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager interrupted
   [junit4]   2> 3973484 WARN  (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_B98C80A4FD974A51-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_B98C80A4FD974A51-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:34162) [    ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1739871814-127.0.0.1-1515945419444 (Datanode Uuid b0e7fb84-5c02-4f90-b605-9723b88c367f) service to lucene2-us-west.apache.org/127.0.0.1:34162
   [junit4]   2> 3973498 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[B98C80A4FD974A51]-worker) [    ] o.m.log Stopped HttpServer2$[hidden email]:0
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.2/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_B98C80A4FD974A51-001
   [junit4]   2> Jan 14, 2018 4:20:23 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 34 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=16845, maxDocsPerChunk=9, blockSize=499), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=16845, blockSize=499)), sim=RandomSimilarity(queryNorm=false): {}, locale=ar-SY, timezone=Africa/Monrovia
   [junit4]   2> NOTE: Linux 4.4.0-104-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=2,free=163844920,total=529006592
   [junit4]   2> NOTE: All tests run in this JVM: [CloudMLTQParserTest, TestDynamicLoading, MoveReplicaHDFSTest, TestFieldResource, TestComplexPhraseLeadingWildcard, DeleteStatusTest, SegmentsInfoRequestHandlerTest, OverseerCollectionConfigSetProcessorTest, TestCSVResponseWriter, TestDistributedMissingSort, TestInPlaceUpdatesDistrib, FileBasedSpellCheckerTest, RecoveryZkTest, AutoAddReplicasIntegrationTest, DeleteNodeTest, InfixSuggestersTest, SystemInfoHandlerTest, TestConfigOverlay, TestConfigReload, AnalyticsMergeStrategyTest, TestFieldSortValues, SuggesterWFSTTest, JvmMetricsTest, TestRealTimeGet, TestSubQueryTransformerDistrib, TestReplicationHandlerBackup, ReturnFieldsTest, MetricsHandlerTest, IndexSchemaTest, TestFuzzyAnalyzedSuggestions, TestHashPartitioner, CoreAdminRequestStatusTest, TestFieldCacheReopen, HdfsWriteToMultipleCollectionsTest, DisMaxRequestHandlerTest, TestLFUCache, TestSystemIdResolver, CoreAdminHandlerTest, TestNoOpRegenerator, OverseerStatusTest, UUIDFieldTest, TestSolrCloudWithHadoopAuthPlugin, FieldMutatingUpdateProcessorTest, AddBlockUpdateTest, DirectSolrConnectionTest, SolrCloudExampleTest, TestInitParams, TestRequestForwarding, TestReloadDeadlock, DistribJoinFromCollectionTest, TestNumericTerms32, CollectionsAPIAsyncDistributedZkTest, TimeRoutedAliasUpdateProcessorTest, SmileWriterTest, HdfsCollectionsAPIDistributedZkTest, TestDynamicFieldCollectionResource, ComputePlanActionTest, HighlighterTest, TestDistribDocBasedVersion, TestSQLHandler, SpatialHeatmapFacetsTest, TestLegacyField, AliasIntegrationTest, LeaderElectionContextKeyTest, TestMaxScoreQueryParser, TimeZoneUtilsTest, TestCoreDiscovery, AsyncCallRequestStatusResponseTest, QueryElevationComponentTest, LeaderElectionTest, FieldAnalysisRequestHandlerTest, SearchRateTriggerTest, TestDocBasedVersionConstraints, EnumFieldTest, TestReload, TestSolrFieldCacheBean, StatsComponentTest, SuggesterTest, TestExportWriter, TestSolrCoreProperties, CdcrBootstrapTest, SharedFSAutoReplicaFailoverTest]
   [junit4] Completed [352/751 (2!)] on J2 in 1413.27s, 1 test, 1 failure <<< FAILURES!

[...truncated 48715 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]