[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 1 - Unstable

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 1 - Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/1/

3 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([F025D098B63D7EEE:7871EF4218C11316]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
8 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:     1) Thread[id=14992, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    2) Thread[id=15093, name=StoppableIndexingThread-SendThread(127.0.0.1:50722), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    3) Thread[id=15101, name=zkCallback-3010-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=15276, name=zkCallback-3010-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=15094, name=StoppableIndexingThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    6) Thread[id=14993, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    7) Thread[id=15274, name=zkCallback-3010-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    8) Thread[id=15275, name=zkCallback-3010-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:
   1) Thread[id=14992, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   2) Thread[id=15093, name=StoppableIndexingThread-SendThread(127.0.0.1:50722), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   3) Thread[id=15101, name=zkCallback-3010-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=15276, name=zkCallback-3010-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=15094, name=StoppableIndexingThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   6) Thread[id=14993, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   7) Thread[id=15274, name=zkCallback-3010-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=15275, name=zkCallback-3010-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([F025D098B63D7EEE]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=14992, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]         at org.apache.solr.common.util.JavaBinCodec.writeStr(JavaBinCodec.java:840)         at org.apache.solr.common.util.JavaBinCodec.writePrimitive(JavaBinCodec.java:932)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)    2) Thread[id=15093, name=StoppableIndexingThread-SendThread(127.0.0.1:50722), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    3) Thread[id=15101, name=zkCallback-3010-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=15276, name=zkCallback-3010-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=14993, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]         at org.apache.solr.common.util.JavaBinCodec.writeExternString(JavaBinCodec.java:1072)         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:615)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)    6) Thread[id=15274, name=zkCallback-3010-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=15275, name=zkCallback-3010-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=14992, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
        at org.apache.solr.common.util.JavaBinCodec.writeStr(JavaBinCodec.java:840)
        at org.apache.solr.common.util.JavaBinCodec.writePrimitive(JavaBinCodec.java:932)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
        at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   2) Thread[id=15093, name=StoppableIndexingThread-SendThread(127.0.0.1:50722), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   3) Thread[id=15101, name=zkCallback-3010-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=15276, name=zkCallback-3010-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=14993, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
        at org.apache.solr.common.util.JavaBinCodec.writeExternString(JavaBinCodec.java:1072)
        at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:615)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
        at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   6) Thread[id=15274, name=zkCallback-3010-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=15275, name=zkCallback-3010-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([F025D098B63D7EEE]:0)




Build Log:
[...truncated 12219 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/init-core-data-001
   [junit4]   2> 1565699 WARN  (SUITE-ChaosMonkeySafeLeaderTest-seed#[F025D098B63D7EEE]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
   [junit4]   2> 1565700 INFO  (SUITE-ChaosMonkeySafeLeaderTest-seed#[F025D098B63D7EEE]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1565701 INFO  (SUITE-ChaosMonkeySafeLeaderTest-seed#[F025D098B63D7EEE]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1565702 INFO  (SUITE-ChaosMonkeySafeLeaderTest-seed#[F025D098B63D7EEE]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1565706 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1565707 INFO  (Thread-4662) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1565707 INFO  (Thread-4662) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1565709 ERROR (Thread-4662) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1565807 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.ZkTestServer start zk server on port:50722
   [junit4]   2> 1565822 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1565825 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1565827 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1565829 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1565831 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1565832 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1565834 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1565836 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1565838 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1565840 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1565842 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1565845 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 1565990 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1565992 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@42dc30aa{/,null,AVAILABLE}
   [junit4]   2> 1565992 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3bc5492c{HTTP/1.1,[http/1.1]}{127.0.0.1:46276}
   [junit4]   2> 1565992 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.e.j.s.Server Started @1569344ms
   [junit4]   2> 1565993 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=46276, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/control-001/cores}
   [junit4]   2> 1565994 ERROR (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1565994 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1565994 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1565994 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1565994 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-05T03:31:10.936Z
   [junit4]   2> 1565997 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1565997 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/control-001/solr.xml
   [junit4]   2> 1566003 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@29d9017c, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1566007 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50722/solr
   [junit4]   2> 1566130 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:46276_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1566131 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:46276_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46276_
   [junit4]   2> 1566132 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:46276_    ] o.a.s.c.Overseer Overseer (id=98253240444518404-127.0.0.1:46276_-n_0000000000) starting
   [junit4]   2> 1566140 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:46276_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46276_
   [junit4]   2> 1566143 INFO  (zkCallback-2906-thread-1-processing-n:127.0.0.1:46276_) [n:127.0.0.1:46276_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1566225 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:46276_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29d9017c
   [junit4]   2> 1566231 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:46276_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29d9017c
   [junit4]   2> 1566232 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:46276_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29d9017c
   [junit4]   2> 1566233 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:46276_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/control-001/cores
   [junit4]   2> 1566255 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1566256 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:50722/solr ready
   [junit4]   2> 1566258 INFO  (qtp1675720877-14478) [n:127.0.0.1:46276_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:46276_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1566260 INFO  (OverseerThreadFactory-4437-thread-1-processing-n:127.0.0.1:46276_) [n:127.0.0.1:46276_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 1566365 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1566365 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1566470 INFO  (zkCallback-2906-thread-1-processing-n:127.0.0.1:46276_) [n:127.0.0.1:46276_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1567382 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1567398 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 1567496 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1567512 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 1567513 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29d9017c
   [junit4]   2> 1567513 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1567513 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 1567517 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=42, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=0.3837890625, floorSegmentMB=0.3408203125, forceMergeDeletesPctAllowed=25.965269595019116, segmentsPerTier=15.0, maxCFSSegmentSizeMB=2.072265625, noCFSRatio=0.0
   [junit4]   2> 1567528 WARN  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1567580 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1567580 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1567581 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
   [junit4]   2> 1567581 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1567583 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@6eb698b
   [junit4]   2> 1567583 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4b13aad3[control_collection_shard1_replica_n1] main]
   [junit4]   2> 1567585 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1567585 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1567586 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1567587 INFO  (searcherExecutor-4440-thread-1-processing-n:127.0.0.1:46276_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@4b13aad3[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1567587 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572051849082568704
   [junit4]   2> 1567593 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1567593 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1567593 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46276/control_collection_shard1_replica_n1/
   [junit4]   2> 1567594 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1567594 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:46276/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 1567594 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1567604 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46276/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 1567706 INFO  (zkCallback-2906-thread-1-processing-n:127.0.0.1:46276_) [n:127.0.0.1:46276_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1567755 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1567757 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1392
   [junit4]   2> 1567760 INFO  (qtp1675720877-14478) [n:127.0.0.1:46276_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1567860 INFO  (zkCallback-2906-thread-1-processing-n:127.0.0.1:46276_) [n:127.0.0.1:46276_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1568262 INFO  (OverseerCollectionConfigSetProcessor-98253240444518404-127.0.0.1:46276_-n_0000000000) [n:127.0.0.1:46276_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1568761 INFO  (qtp1675720877-14478) [n:127.0.0.1:46276_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:46276_&wt=javabin&version=2} status=0 QTime=2503
   [junit4]   2> 1568766 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1568767 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:50722/solr ready
   [junit4]   2> 1568768 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1568768 INFO  (qtp1675720877-14475) [n:127.0.0.1:46276_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=3&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1568771 INFO  (OverseerThreadFactory-4437-thread-2-processing-n:127.0.0.1:46276_) [n:127.0.0.1:46276_    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 1568771 WARN  (OverseerThreadFactory-4437-thread-2-processing-n:127.0.0.1:46276_) [n:127.0.0.1:46276_    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 1568977 INFO  (qtp1675720877-14475) [n:127.0.0.1:46276_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1568977 INFO  (qtp1675720877-14475) [n:127.0.0.1:46276_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=3&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=208
   [junit4]   2> 1569100 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/shard-1-001 of type NRT
   [junit4]   2> 1569101 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1569102 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@14501b32{/,null,AVAILABLE}
   [junit4]   2> 1569102 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@30dfc323{HTTP/1.1,[http/1.1]}{127.0.0.1:33146}
   [junit4]   2> 1569103 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.e.j.s.Server Started @1572454ms
   [junit4]   2> 1569103 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=33146, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/shard-1-001/cores}
   [junit4]   2> 1569104 ERROR (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1569104 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1569104 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1569104 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1569104 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-05T03:31:14.046Z
   [junit4]   2> 1569107 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1569107 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/shard-1-001/solr.xml
   [junit4]   2> 1569112 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@29d9017c, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1569115 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50722/solr
   [junit4]   2> 1569126 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:33146_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1569128 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:33146_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1569131 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:33146_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33146_
   [junit4]   2> 1569132 INFO  (zkCallback-2918-thread-1-processing-n:127.0.0.1:33146_) [n:127.0.0.1:33146_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1569132 INFO  (zkCallback-2906-thread-1-processing-n:127.0.0.1:46276_) [n:127.0.0.1:46276_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1569132 INFO  (zkCallback-2913-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1569195 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:33146_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29d9017c
   [junit4]   2> 1569201 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:33146_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29d9017c
   [junit4]   2> 1569201 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:33146_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29d9017c
   [junit4]   2> 1569202 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:33146_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/shard-1-001/cores
   [junit4]   2> 1569227 INFO  (qtp1675720877-14476) [n:127.0.0.1:46276_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:33146_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1569230 INFO  (OverseerCollectionConfigSetProcessor-98253240444518404-127.0.0.1:46276_-n_0000000000) [n:127.0.0.1:46276_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1569231 INFO  (OverseerThreadFactory-4437-thread-3-processing-n:127.0.0.1:46276_) [n:127.0.0.1:46276_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:33146_ for creating new replica
   [junit4]   2> 1569232 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1569233 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1569338 INFO  (zkCallback-2918-thread-1-processing-n:127.0.0.1:33146_) [n:127.0.0.1:33146_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1570269 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1570285 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema [collection1_shard2_replica_n0] Schema name=test
   [junit4]   2> 1570382 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1570403 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n0' using configuration from collection collection1, trusted=true
   [junit4]   2> 1570403 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n0' (registry 'solr.core.collection1.shard2.replica_n0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29d9017c
   [junit4]   2> 1570404 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1570404 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore [[collection1_shard2_replica_n0] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/shard-1-001/cores/collection1_shard2_replica_n0], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/shard-1-001/cores/collection1_shard2_replica_n0/data/]
   [junit4]   2> 1570408 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=42, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=0.3837890625, floorSegmentMB=0.3408203125, forceMergeDeletesPctAllowed=25.965269595019116, segmentsPerTier=15.0, maxCFSSegmentSizeMB=2.072265625, noCFSRatio=0.0
   [junit4]   2> 1570422 WARN  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1570496 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1570497 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1570498 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
   [junit4]   2> 1570498 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1570500 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@68964a9f
   [junit4]   2> 1570500 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.SolrIndexSearcher Opening [Searcher@38018456[collection1_shard2_replica_n0] main]
   [junit4]   2> 1570501 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1570502 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1570502 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1570503 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572051852140216320
   [junit4]   2> 1570506 INFO  (searcherExecutor-4451-thread-1-processing-n:127.0.0.1:33146_ x:collection1_shard2_replica_n0 s:shard2 c:collection1) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore [collection1_shard2_replica_n0] Registered new searcher Searcher@38018456[collection1_shard2_replica_n0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1570510 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1570510 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1570510 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33146/collection1_shard2_replica_n0/
   [junit4]   2> 1570510 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1570510 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy http://127.0.0.1:33146/collection1_shard2_replica_n0/ has no replicas
   [junit4]   2> 1570510 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1570519 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33146/collection1_shard2_replica_n0/ shard2
   [junit4]   2> 1570627 INFO  (zkCallback-2918-thread-1-processing-n:127.0.0.1:33146_) [n:127.0.0.1:33146_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1570676 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1570680 INFO  (qtp1398025774-14519) [n:127.0.0.1:33146_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1447
   [junit4]   2> 1570686 INFO  (qtp1675720877-14476) [n:127.0.0.1:46276_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:33146_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1458
   [junit4]   2> 1570786 INFO  (zkCallback-2918-thread-1-processing-n:127.0.0.1:33146_) [n:127.0.0.1:33146_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1570846 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/shard-2-001 of type NRT
   [junit4]   2> 1570846 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1570848 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5a7b5dcc{/,null,AVAILABLE}
   [junit4]   2> 1570848 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2ee3cf8c{HTTP/1.1,[http/1.1]}{127.0.0.1:35194}
   [junit4]   2> 1570848 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.e.j.s.Server Started @1574200ms
   [junit4]   2> 1570848 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=35194, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/shard-2-001/cores}
   [junit4]   2> 1570849 ERROR (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1570849 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1570849 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1570849 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1570849 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-05T03:31:15.791Z
   [junit4]   2> 1570859 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1570859 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/shard-2-001/solr.xml
   [junit4]   2> 1570864 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@29d9017c, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1570875 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50722/solr
   [junit4]   2> 1570917 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:35194_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1570920 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:35194_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1570922 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:35194_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35194_
   [junit4]   2> 1570926 INFO  (zkCallback-2906-thread-1-processing-n:127.0.0.1:46276_) [n:127.0.0.1:46276_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1570926 INFO  (zkCallback-2913-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1570926 INFO  (zkCallback-2918-thread-1-processing-n:127.0.0.1:33146_) [n:127.0.0.1:33146_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1570927 INFO  (zkCallback-2924-thread-1-processing-n:127.0.0.1:35194_) [n:127.0.0.1:35194_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1571092 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:35194_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29d9017c
   [junit4]   2> 1571099 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:35194_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29d9017c
   [junit4]   2> 1571099 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:35194_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29d9017c
   [junit4]   2> 1571101 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[F025D098B63D7EEE]) [n:127.0.0.1:35194_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/shard-2-001/cores
   [junit4]   2> 1571148 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:35194_&action=ADDREPLICA&collection=collection1&shard=shard3&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1571160 INFO  (OverseerCollectionConfigSetProcessor-98253240444518404-127.0.0.1:46276_-n_0000000000) [n:127.0.0.1:46276_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1571161 INFO  (OverseerThreadFactory-4437-thread-4-processing-n:127.0.0.1:46276_) [n:127.0.0.1:46276_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:35194_ for creating new replica
   [junit4]   2> 1571163 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard3_replica_n0&action=CREATE&collection=collection1&shard=shard3&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1571164 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1571267 INFO  (zkCallback-2918-thread-1-processing-n:127.0.0.1:33146_) [n:127.0.0.1:33146_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1571267 INFO  (zkCallback-2924-thread-1-processing-n:127.0.0.1:35194_) [n:127.0.0.1:35194_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1572187 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1572201 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.s.IndexSchema [collection1_shard3_replica_n0] Schema name=test
   [junit4]   2> 1572301 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1572325 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard3_replica_n0' using configuration from collection collection1, trusted=true
   [junit4]   2> 1572325 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard3.replica_n0' (registry 'solr.core.collection1.shard3.replica_n0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29d9017c
   [junit4]   2> 1572325 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1572325 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.SolrCore [[collection1_shard3_replica_n0] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/shard-2-001/cores/collection1_shard3_replica_n0], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_F025D098B63D7EEE-001/shard-2-001/cores/collection1_shard3_replica_n0/data/]
   [junit4]   2> 1572329 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=42, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=0.3837890625, floorSegmentMB=0.3408203125, forceMergeDeletesPctAllowed=25.965269595019116, segmentsPerTier=15.0, maxCFSSegmentSizeMB=2.072265625, noCFSRatio=0.0
   [junit4]   2> 1572354 WARN  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1572410 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1572411 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1572412 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
   [junit4]   2> 1572412 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1572413 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@a0572f
   [junit4]   2> 1572414 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.s.SolrIndexSearcher Opening [Searcher@17c8bb6b[collection1_shard3_replica_n0] main]
   [junit4]   2> 1572416 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1572416 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1572417 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1572418 INFO  (searcherExecutor-4462-thread-1-processing-n:127.0.0.1:35194_ x:collection1_shard3_replica_n0 s:shard3 c:collection1) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.SolrCore [collection1_shard3_replica_n0] Registered new searcher Searcher@17c8bb6b[collection1_shard3_replica_n0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1572418 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572051854148239360
   [junit4]   2> 1572425 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1572425 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1572425 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:35194/collection1_shard3_replica_n0/
   [junit4]   2> 1572425 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1572425 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.SyncStrategy http://127.0.0.1:35194/collection1_shard3_replica_n0/ has no replicas
   [junit4]   2> 1572425 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1572433 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:35194/collection1_shard3_replica_n0/ shard3
   [junit4]   2> 1572536 INFO  (zkCallback-2918-thread-1-processing-n:127.0.0.1:33146_) [n:127.0.0.1:33146_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1572536 INFO  (zkCallback-2924-thread-1-processing-n:127.0.0.1:35194_) [n:127.0.0.1:35194_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1572584 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1572587 INFO  (qtp424011568-14549) [n:127.0.0.1:35194_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard3_replica_n0&action=CREATE&collection=collection1&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1423
   [junit4]   2> 1572593 INFO  (qtp1675720877-14480) [n:127.0.0.1:46276_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:35194_&action=ADDREPLICA&collection=collection1&shard=shard3&type=NRT&wt=javabin&version=2} status=0 QTime=1444
   [junit4]   2> 1572692 INFO  (zkCallback-2918-thread-1-processing-n:127.0.0.1:33146_) [n:127.0.0.1:33146_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1572692 INFO  (zkCallback-2924-thread-1-processing-n:127.0.0.1:35194_) [n:127.0.0.1:35194_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1572721 INFO  (TEST-Cha

[...truncated too long message...]

t4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
   [junit4]    >         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
   [junit4]    >         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   [junit4]    >    2) Thread[id=15093, name=StoppableIndexingThread-SendThread(127.0.0.1:50722), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   [junit4]    >    3) Thread[id=15101, name=zkCallback-3010-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    4) Thread[id=15276, name=zkCallback-3010-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    5) Thread[id=14993, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeExternString(JavaBinCodec.java:1072)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:615)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
   [junit4]    >         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
   [junit4]    >         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   [junit4]    >    6) Thread[id=15274, name=zkCallback-3010-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    7) Thread[id=15275, name=zkCallback-3010-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([F025D098B63D7EEE]:0)
   [junit4] Completed [282/726 (1!)] on J2 in 157.13s, 1 test, 1 failure, 2 errors <<< FAILURES!

[...truncated 44740 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 2 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/2/

1 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test

Error Message:
Could not find collection : unloadcollection

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : unloadcollection
        at __randomizedtesting.SeedInfo.seed([4326C8729C267E15:CB72F7A832DA13ED]:0)
        at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:194)
        at org.apache.solr.cloud.UnloadDistributedZkTest.testCoreUnloadAndLeaders(UnloadDistributedZkTest.java:177)
        at org.apache.solr.cloud.UnloadDistributedZkTest.test(UnloadDistributedZkTest.java:67)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13241 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/init-core-data-001
   [junit4]   2> 3711456 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[4326C8729C267E15]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 3711458 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[4326C8729C267E15]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 3711458 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[4326C8729C267E15]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /_/
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 3711497 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[4326C8729C267E15]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 3711511 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[4326C8729C267E15]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3711512 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[4326C8729C267E15]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3711525 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[4326C8729C267E15]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_54608_hdfs____.e1srkn/webapp
   [junit4]   2> 3711899 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[4326C8729C267E15]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:54608
   [junit4]   2> 3711949 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[4326C8729C267E15]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3711950 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[4326C8729C267E15]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3711962 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[4326C8729C267E15]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_35753_datanode____.4bqnty/webapp
   [junit4]   2> 3712267 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[4326C8729C267E15]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35753
   [junit4]   2> 3712324 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[4326C8729C267E15]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3712325 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[4326C8729C267E15]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3712338 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[4326C8729C267E15]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_37894_datanode____.ykrs4q/webapp
   [junit4]   2> 3712407 INFO  (IPC Server handler 3 on 47974) [    ] BlockStateChange BLOCK* processReport: from storage DS-299bbd71-1ade-4256-897c-300d52538ec5 node DatanodeRegistration(127.0.0.1:42130, datanodeUuid=fa719570-f52b-4931-9dfd-0c730f2636ab, infoPort=57812, infoSecurePort=0, ipcPort=42511, storageInfo=lv=-56;cid=testClusterID;nsid=969051017;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 3712407 INFO  (IPC Server handler 3 on 47974) [    ] BlockStateChange BLOCK* processReport: from storage DS-adebf495-c297-4485-9a3d-0027f4e863dd node DatanodeRegistration(127.0.0.1:42130, datanodeUuid=fa719570-f52b-4931-9dfd-0c730f2636ab, infoPort=57812, infoSecurePort=0, ipcPort=42511, storageInfo=lv=-56;cid=testClusterID;nsid=969051017;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 3712735 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[4326C8729C267E15]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37894
   [junit4]   2> 3712829 INFO  (IPC Server handler 8 on 47974) [    ] BlockStateChange BLOCK* processReport: from storage DS-5e431bdb-8ed3-4f73-8141-a29fe1a40e31 node DatanodeRegistration(127.0.0.1:38456, datanodeUuid=f36986f0-0e14-4f94-a043-6b49f231762d, infoPort=40426, infoSecurePort=0, ipcPort=57728, storageInfo=lv=-56;cid=testClusterID;nsid=969051017;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 3712829 INFO  (IPC Server handler 8 on 47974) [    ] BlockStateChange BLOCK* processReport: from storage DS-f5472fb4-5bd7-4416-b975-5a1d79348247 node DatanodeRegistration(127.0.0.1:38456, datanodeUuid=f36986f0-0e14-4f94-a043-6b49f231762d, infoPort=40426, infoSecurePort=0, ipcPort=57728, storageInfo=lv=-56;cid=testClusterID;nsid=969051017;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 3712902 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3712902 INFO  (Thread-104901) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3712902 INFO  (Thread-104901) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3712904 ERROR (Thread-104901) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3713002 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.ZkTestServer start zk server on port:37313
   [junit4]   2> 3713014 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3713016 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 3713017 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3713019 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3713020 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3713022 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3713023 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3713025 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3713026 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3713027 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3713029 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3713030 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 3713175 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3713176 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2959ce14{/_,null,AVAILABLE}
   [junit4]   2> 3713177 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@a522dc{HTTP/1.1,[http/1.1]}{127.0.0.1:56377}
   [junit4]   2> 3713177 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.e.j.s.Server Started @3716270ms
   [junit4]   2> 3713177 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:47974/hdfs__localhost_47974__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.x_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001_tempDir-002_control_data, replicaType=NRT, hostContext=/_, hostPort=56377, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/control-001/cores}
   [junit4]   2> 3713177 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3713177 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 3713177 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3713177 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 3713177 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-06T04:28:36.930Z
   [junit4]   2> 3713180 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3713180 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/control-001/solr.xml
   [junit4]   2> 3713197 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3713202 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37313/solr
   [junit4]   2> 3713298 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:56377__    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3713299 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:56377__    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:56377__
   [junit4]   2> 3713300 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:56377__    ] o.a.s.c.Overseer Overseer (id=98259128592367620-127.0.0.1:56377__-n_0000000000) starting
   [junit4]   2> 3713307 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:56377__    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56377__
   [junit4]   2> 3713309 INFO  (zkCallback-4288-thread-1-processing-n:127.0.0.1:56377__) [n:127.0.0.1:56377__    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3713387 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:56377__    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7
   [junit4]   2> 3713392 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:56377__    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7
   [junit4]   2> 3713392 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:56377__    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7
   [junit4]   2> 3713394 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:56377__    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/control-001/cores
   [junit4]   2> 3713412 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3713413 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37313/solr ready
   [junit4]   2> 3713414 INFO  (qtp297630363-126927) [n:127.0.0.1:56377__    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:56377__&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3713416 INFO  (OverseerThreadFactory-8743-thread-1-processing-n:127.0.0.1:56377__) [n:127.0.0.1:56377__    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 3713521 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3713521 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3713625 INFO  (zkCallback-4288-thread-1-processing-n:127.0.0.1:56377__) [n:127.0.0.1:56377__    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3714536 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3714550 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 3714664 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3714681 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 3714681 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7
   [junit4]   2> 3714681 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:47974/solr_hdfs_home
   [junit4]   2> 3714681 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3714681 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3714682 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[hdfs://localhost:47974/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 3714683 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:47974/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 3714691 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3714691 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3714692 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3714698 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3714698 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:47974/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 3714724 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:47974/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 3714732 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3714732 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3714732 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3714738 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3714738 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=33, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.291906120820587]
   [junit4]   2> 3714756 INFO  (IPC Server handler 3 on 47974) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38456 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-adebf495-c297-4485-9a3d-0027f4e863dd:NORMAL:127.0.0.1:42130|RBW], ReplicaUC[[DISK]DS-5e431bdb-8ed3-4f73-8141-a29fe1a40e31:NORMAL:127.0.0.1:38456|RBW]]} size 0
   [junit4]   2> 3714758 INFO  (IPC Server handler 5 on 47974) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42130 is added to blk_1073741825_1001 size 72
   [junit4]   2> 3714763 WARN  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3714833 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 3714833 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3714833 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 3714843 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3714843 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3714848 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=87.17578125, floorSegmentMB=1.4677734375, forceMergeDeletesPctAllowed=24.808359503547322, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2595128315016833
   [junit4]   2> 3714855 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4804450c[control_collection_shard1_replica_n1] main]
   [junit4]   2> 3714856 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3714857 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3714858 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3714859 INFO  (searcherExecutor-8746-thread-1-processing-n:127.0.0.1:56377__ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@4804450c[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3714859 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572146059528896512
   [junit4]   2> 3714871 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3714871 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3714871 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:56377/_/control_collection_shard1_replica_n1/
   [junit4]   2> 3714871 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3714871 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:56377/_/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 3714871 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3714877 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:56377/_/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 3714979 INFO  (zkCallback-4288-thread-1-processing-n:127.0.0.1:56377__) [n:127.0.0.1:56377__    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3715028 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3715030 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1509
   [junit4]   2> 3715034 INFO  (qtp297630363-126927) [n:127.0.0.1:56377__    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 3715134 INFO  (zkCallback-4288-thread-1-processing-n:127.0.0.1:56377__) [n:127.0.0.1:56377__    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3715418 INFO  (OverseerCollectionConfigSetProcessor-98259128592367620-127.0.0.1:56377__-n_0000000000) [n:127.0.0.1:56377__    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3716034 INFO  (qtp297630363-126927) [n:127.0.0.1:56377__    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:56377__&wt=javabin&version=2} status=0 QTime=2619
   [junit4]   2> 3716039 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3716040 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37313/solr ready
   [junit4]   2> 3716040 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3716041 INFO  (qtp297630363-126924) [n:127.0.0.1:56377__    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3716043 INFO  (OverseerThreadFactory-8743-thread-2-processing-n:127.0.0.1:56377__) [n:127.0.0.1:56377__    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 3716044 WARN  (OverseerThreadFactory-8743-thread-2-processing-n:127.0.0.1:56377__) [n:127.0.0.1:56377__    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 3716248 INFO  (qtp297630363-126924) [n:127.0.0.1:56377__    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 3716248 INFO  (qtp297630363-126924) [n:127.0.0.1:56377__    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=206
   [junit4]   2> 3716377 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/shard-1-001 of type NRT
   [junit4]   2> 3716377 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3716379 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@43fe3b40{/_,null,AVAILABLE}
   [junit4]   2> 3716379 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5b9244cc{HTTP/1.1,[http/1.1]}{127.0.0.1:34709}
   [junit4]   2> 3716379 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.e.j.s.Server Started @3719473ms
   [junit4]   2> 3716379 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:47974/hdfs__localhost_47974__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.x_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/_, hostPort=34709, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/shard-1-001/cores}
   [junit4]   2> 3716379 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3716380 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 3716380 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3716380 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 3716380 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-06T04:28:40.133Z
   [junit4]   2> 3716382 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3716382 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/shard-1-001/solr.xml
   [junit4]   2> 3716388 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3716391 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37313/solr
   [junit4]   2> 3716399 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:34709__    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3716402 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:34709__    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3716404 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:34709__    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34709__
   [junit4]   2> 3716405 INFO  (zkCallback-4288-thread-1-processing-n:127.0.0.1:56377__) [n:127.0.0.1:56377__    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3716405 INFO  (zkCallback-4295-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3716406 INFO  (zkCallback-4300-thread-1-processing-n:127.0.0.1:34709__) [n:127.0.0.1:34709__    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3716536 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:34709__    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7
   [junit4]   2> 3716542 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:34709__    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7
   [junit4]   2> 3716542 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:34709__    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7
   [junit4]   2> 3716543 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:34709__    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/shard-1-001/cores
   [junit4]   2> 3716566 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:34709__&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3716568 INFO  (OverseerThreadFactory-8743-thread-3-processing-n:127.0.0.1:56377__) [n:127.0.0.1:56377__    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:34709__ for creating new replica
   [junit4]   2> 3716568 INFO  (OverseerCollectionConfigSetProcessor-98259128592367620-127.0.0.1:56377__-n_0000000000) [n:127.0.0.1:56377__    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3716570 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3716570 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3717584 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3717600 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema [collection1_shard2_replica_n0] Schema name=test
   [junit4]   2> 3717726 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3717746 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n0' using configuration from collection collection1, trusted=true
   [junit4]   2> 3717747 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n0' (registry 'solr.core.collection1.shard2.replica_n0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7
   [junit4]   2> 3717747 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:47974/solr_hdfs_home
   [junit4]   2> 3717747 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3717747 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3717747 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore [[collection1_shard2_replica_n0] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/shard-1-001/cores/collection1_shard2_replica_n0], dataDir=[hdfs://localhost:47974/solr_hdfs_home/collection1/core_node1/data/]
   [junit4]   2> 3717749 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:47974/solr_hdfs_home/collection1/core_node1/data/snapshot_metadata
   [junit4]   2> 3717757 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3717757 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3717757 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3717761 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3717761 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:47974/solr_hdfs_home/collection1/core_node1/data
   [junit4]   2> 3717777 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:47974/solr_hdfs_home/collection1/core_node1/data/index
   [junit4]   2> 3717785 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3717785 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3717785 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3717790 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3717790 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=33, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.291906120820587]
   [junit4]   2> 3717810 INFO  (IPC Server handler 5 on 47974) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42130 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5e431bdb-8ed3-4f73-8141-a29fe1a40e31:NORMAL:127.0.0.1:38456|RBW], ReplicaUC[[DISK]DS-adebf495-c297-4485-9a3d-0027f4e863dd:NORMAL:127.0.0.1:42130|FINALIZED]]} size 0
   [junit4]   2> 3717811 INFO  (IPC Server handler 6 on 47974) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38456 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-adebf495-c297-4485-9a3d-0027f4e863dd:NORMAL:127.0.0.1:42130|FINALIZED], ReplicaUC[[DISK]DS-f5472fb4-5bd7-4416-b975-5a1d79348247:NORMAL:127.0.0.1:38456|FINALIZED]]} size 0
   [junit4]   2> 3717816 WARN  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3717866 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 3717867 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3717867 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 3717877 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3717877 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3717879 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=87.17578125, floorSegmentMB=1.4677734375, forceMergeDeletesPctAllowed=24.808359503547322, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2595128315016833
   [junit4]   2> 3717883 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.SolrIndexSearcher Opening [Searcher@185d196e[collection1_shard2_replica_n0] main]
   [junit4]   2> 3717884 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3717885 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3717885 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3717887 INFO  (searcherExecutor-8757-thread-1-processing-n:127.0.0.1:34709__ x:collection1_shard2_replica_n0 s:shard2 c:collection1) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore [collection1_shard2_replica_n0] Registered new searcher Searcher@185d196e[collection1_shard2_replica_n0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3717887 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572146062703984640
   [junit4]   2> 3717893 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3717893 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3717893 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:34709/_/collection1_shard2_replica_n0/
   [junit4]   2> 3717893 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3717893 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy http://127.0.0.1:34709/_/collection1_shard2_replica_n0/ has no replicas
   [junit4]   2> 3717894 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3717897 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:34709/_/collection1_shard2_replica_n0/ shard2
   [junit4]   2> 3718048 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3718050 INFO  (qtp1620848425-126978) [n:127.0.0.1:34709__ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1480
   [junit4]   2> 3718052 INFO  (qtp297630363-126925) [n:127.0.0.1:56377__    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:34709__&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1486
   [junit4]   2> 3718188 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/shard-2-001 of type NRT
   [junit4]   2> 3718189 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3718193 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4be5d278{/_,null,AVAILABLE}
   [junit4]   2> 3718193 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@48de5c53{HTTP/1.1,[http/1.1]}{127.0.0.1:47668}
   [junit4]   2> 3718193 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.e.j.s.Server Started @3721287ms
   [junit4]   2> 3718193 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:47974/hdfs__localhost_47974__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.x_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001_tempDir-002_jetty2, solrconfig=solrconfig.xml, hostContext=/_, hostPort=47668, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/shard-2-001/cores}
   [junit4]   2> 3718194 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3718194 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 3718194 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3718194 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 3718194 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-06T04:28:41.947Z
   [junit4]   2> 3718197 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3718197 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/shard-2-001/solr.xml
   [junit4]   2> 3718202 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3718205 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37313/solr
   [junit4]   2> 3718214 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:47668__    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3718216 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:47668__    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3718218 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:47668__    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47668__
   [junit4]   2> 3718220 INFO  (zkCallback-4300-thread-1-processing-n:127.0.0.1:34709__) [n:127.0.0.1:34709__    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3718220 INFO  (zkCallback-4295-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3718220 INFO  (zkCallback-4288-thread-1-processing-n:127.0.0.1:56377__) [n:127.0.0.1:56377__    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3718228 INFO  (zkCallback-4306-thread-1-processing-n:127.0.0.1:47668__) [n:127.0.0.1:47668__    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3718354 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:47668__    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7
   [junit4]   2> 3718361 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:47668__    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7
   [junit4]   2> 3718361 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:47668__    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7
   [junit4]   2> 3718362 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:47668__    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/shard-2-001/cores
   [junit4]   2> 3718392 INFO  (qtp297630363-126927) [n:127.0.0.1:56377__    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:47668__&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3718395 INFO  (OverseerThreadFactory-8743-thread-4-processing-n:127.0.0.1:56377__) [n:127.0.0.1:56377__    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:47668__ for creating new replica
   [junit4]   2> 3718395 INFO  (OverseerCollectionConfigSetProcessor-98259128592367620-127.0.0.1:56377__-n_0000000000) [n:127.0.0.1:56377__    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3718397 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n0&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3718397 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3719415 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3719444 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.IndexSchema [collection1_shard1_replica_n0] Schema name=test
   [junit4]   2> 3719568 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3719594 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n0' using configuration from collection collection1, trusted=true
   [junit4]   2> 3719595 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n0' (registry 'solr.core.collection1.shard1.replica_n0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7
   [junit4]   2> 3719595 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:47974/solr_hdfs_home
   [junit4]   2> 3719595 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3719595 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3719595 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrCore [[collection1_shard1_replica_n0] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/shard-2-001/cores/collection1_shard1_replica_n0], dataDir=[hdfs://localhost:47974/solr_hdfs_home/collection1/core_node2/data/]
   [junit4]   2> 3719597 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:47974/solr_hdfs_home/collection1/core_node2/data/snapshot_metadata
   [junit4]   2> 3719607 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3719607 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3719607 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3719612 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3719613 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:47974/solr_hdfs_home/collection1/core_node2/data
   [junit4]   2> 3719631 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:47974/solr_hdfs_home/collection1/core_node2/data/index
   [junit4]   2> 3719640 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3719640 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3719640 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3719646 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3719647 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=33, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.291906120820587]
   [junit4]   2> 3719656 INFO  (IPC Server handler 7 on 47974) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38456 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-adebf495-c297-4485-9a3d-0027f4e863dd:NORMAL:127.0.0.1:42130|RBW], ReplicaUC[[DISK]DS-5e431bdb-8ed3-4f73-8141-a29fe1a40e31:NORMAL:127.0.0.1:38456|FINALIZED]]} size 0
   [junit4]   2> 3719656 INFO  (IPC Server handler 9 on 47974) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42130 is added to blk_1073741827_1003 size 72
   [junit4]   2> 3719660 WARN  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3719720 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 3719720 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3719720 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 3719741 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3719741 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3719744 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=87.17578125, floorSegmentMB=1.4677734375, forceMergeDeletesPctAllowed=24.808359503547322, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2595128315016833
   [junit4]   2> 3719748 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.SolrIndexSearcher Opening [Searcher@128253d8[collection1_shard1_replica_n0] main]
   [junit4]   2> 3719749 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3719750 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3719751 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3719752 INFO  (searcherExecutor-8768-thread-1-processing-n:127.0.0.1:47668__ x:collection1_shard1_replica_n0 s:shard1 c:collection1) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrCore [collection1_shard1_replica_n0] Registered new searcher Searcher@128253d8[collection1_shard1_replica_n0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3719753 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572146064660627456
   [junit4]   2> 3719759 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3719759 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3719759 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:47668/_/collection1_shard1_replica_n0/
   [junit4]   2> 3719759 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3719759 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SyncStrategy http://127.0.0.1:47668/_/collection1_shard1_replica_n0/ has no replicas
   [junit4]   2> 3719759 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3719762 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:47668/_/collection1_shard1_replica_n0/ shard1
   [junit4]   2> 3719913 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3719915 INFO  (qtp686999210-127015) [n:127.0.0.1:47668__ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n0&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1518
   [junit4]   2> 3719917 INFO  (qtp297630363-126927) [n:127.0.0.1:56377__    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:47668__&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1525
   [junit4]   2> 3720046 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/shard-3-001 of type NRT
   [junit4]   2> 3720046 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3720048 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1cbe6529{/_,null,AVAILABLE}
   [junit4]   2> 3720048 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@10e0ec0f{HTTP/1.1,[http/1.1]}{127.0.0.1:39431}
   [junit4]   2> 3720048 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.e.j.s.Server Started @3723142ms
   [junit4]   2> 3720048 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:47974/hdfs__localhost_47974__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.x_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001_tempDir-002_jetty3, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/_, hostPort=39431, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/shard-3-001/cores}
   [junit4]   2> 3720048 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3720049 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 3720049 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3720049 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 3720049 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-06T04:28:43.802Z
   [junit4]   2> 3720052 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3720052 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/shard-3-001/solr.xml
   [junit4]   2> 3720057 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3720060 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37313/solr
   [junit4]   2> 3720069 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:39431__    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 3720071 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:39431__    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3720074 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:39431__    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39431__
   [junit4]   2> 3720075 INFO  (zkCallback-4300-thread-1-processing-n:127.0.0.1:34709__) [n:127.0.0.1:34709__    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3720075 INFO  (zkCallback-4288-thread-1-processing-n:127.0.0.1:56377__) [n:127.0.0.1:56377__    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3720075 INFO  (zkCallback-4306-thread-1-processing-n:127.0.0.1:47668__) [n:127.0.0.1:47668__    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3720075 INFO  (zkCallback-4295-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3720076 INFO  (zkCallback-4312-thread-1-processing-n:127.0.0.1:39431__) [n:127.0.0.1:39431__    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3720163 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:39431__    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7
   [junit4]   2> 3720169 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:39431__    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7
   [junit4]   2> 3720169 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:39431__    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7
   [junit4]   2> 3720171 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[4326C8729C267E15]) [n:127.0.0.1:39431__    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/shard-3-001/cores
   [junit4]   2> 3720220 INFO  (qtp297630363-126928) [n:127.0.0.1:56377__    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:39431__&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3720223 INFO  (OverseerThreadFactory-8743-thread-5-processing-n:127.0.0.1:56377__) [n:127.0.0.1:56377__    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:39431__ for creating new replica
   [junit4]   2> 3720223 INFO  (OverseerCollectionConfigSetProcessor-98259128592367620-127.0.0.1:56377__-n_0000000000) [n:127.0.0.1:56377__    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3720225 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n1&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3720225 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3721239 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3721254 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema [collection1_shard2_replica_n1] Schema name=test
   [junit4]   2> 3721362 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3721382 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 3721382 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n1' (registry 'solr.core.collection1.shard2.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75c43da7
   [junit4]   2> 3721382 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:47974/solr_hdfs_home
   [junit4]   2> 3721382 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3721382 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3721383 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [[collection1_shard2_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001/shard-3-001/cores/collection1_shard2_replica_n1], dataDir=[hdfs://localhost:47974/solr_hdfs_home/collection1/core_node3/data/]
   [junit4]   2> 3721384 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:47974/solr_hdfs_home/collection1/core_node3/data/snapshot_metadata
   [junit4]   2> 3721392 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3721392 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3721392 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3721396 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3721397 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:47974/solr_hdfs_home/collection1/core_node3/data
   [junit4]   2> 3721413 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:47974/solr_hdfs_home/collection1/core_node3/data/index
   [junit4]   2> 3721419 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3721419 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3721419 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3721424 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3721424 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=33, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.291906120820587]
   [junit4]   2> 3721432 INFO  (IPC Server handler 9 on 47974) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38456 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-299bbd71-1ade-4256-897c-300d52538ec5:NORMAL:127.0.0.1:42130|RBW], ReplicaUC[[DISK]DS-f5472fb4-5bd7-4416-b975-5a1d79348247:NORMAL:127.0.0.1:38456|RBW]]} size 0
   [junit4]   2> 3721432 INFO  (IPC Server handler 0 on 47974) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42130 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f5472fb4-5bd7-4416-b975-5a1d79348247:NORMAL:127.0.0.1:38456|RBW], ReplicaUC[[DISK]DS-adebf495-c297-4485-9a3d-0027f4e863dd:NORMAL:127.0.0.1:42130|FINALIZED]]} size 0
   [junit4]   2> 3721436 WARN  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3721486 INFO  (qtp854380947-127052) [n:127.0.0.1:39431__ c:collection1 s

[...truncated too long message...]

nServerInterceptor.java:1595)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2> at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2> at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:226)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:204)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.teardownClass(HdfsUnloadDistributedZkTest.java:46)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> ... 54 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_4326C8729C267E15-001
   [junit4]   2> Jul 06, 2017 4:29:24 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 35 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1924, maxMBSortInHeap=5.0779462663304535, sim=RandomSimilarity(queryNorm=false): {}, locale=hi-IN, timezone=America/Thule
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=2,free=270341768,total=522190848
   [junit4]   2> NOTE: All tests run in this JVM: [TestCollationFieldDocValues, StressHdfsTest, TestSolrCLIRunExample, TestCustomSort, LukeRequestHandlerTest, TestPerFieldSimilarityWithDefaultOverride, OverseerRolesTest, SystemInfoHandlerTest, HdfsLockFactoryTest, TestMinMaxOnMultiValuedField, TestSimpleTrackingShardHandler, TestUseDocValuesAsStored2, TestManagedStopFilterFactory, SuggesterTSTTest, TestMiniSolrCloudCluster, TestClassicSimilarityFactory, TestPayloadCheckQParserPlugin, TestLRUCache, DirectSolrConnectionTest, TestDistributedStatsComponentCardinality, CursorMarkTest, RollingRestartTest, TestManagedResource, DisMaxRequestHandlerTest, CoreAdminHandlerTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestShortCircuitedRequests, TestValueSourceCache, TestPolicyCloud, TestPartialUpdateDeduplication, HdfsRestartWhileUpdatingTest, TestSolrJ, SolrGangliaReporterTest, TestRebalanceLeaders, ResourceLoaderTest, FileBasedSpellCheckerTest, AlternateDirectoryTest, DistributedFacetPivotLargeTest, TestInPlaceUpdatesDistrib, DirectoryFactoryTest, TestBM25SimilarityFactory, ConfigSetsAPITest, LeaderElectionContextKeyTest, SpatialHeatmapFacetsTest, TestPerFieldSimilarity, TestCoreContainer, ScriptEngineTest, ResponseHeaderTest, RankQueryTest, TestDistribIDF, TestPivotHelperCode, ActionThrottleTest, MissingSegmentRecoveryTest, AnalysisAfterCoreReloadTest, ConvertedLegacyTest, CurrencyFieldTypeTest, PrimUtilsTest, ClusterStateUpdateTest, TestRTGBase, ReplaceNodeTest, SOLR749Test, DataDrivenBlockJoinTest, TriLevelCompositeIdRoutingTest, TestImpersonationWithHadoopAuth, TestStressRecovery, TestCorePropertiesReload, NoCacheHeaderTest, TestCollapseQParserPlugin, UninvertDocValuesMergePolicyTest, TestRestoreCore, TestSchemaNameResource, TestHashPartitioner, TestCloudPseudoReturnFields, TestHdfsBackupRestoreCore, TestMacroExpander, CSVRequestHandlerTest, TestNumericRangeQuery64, TestConfigSetImmutable, DistributedIntervalFacetingTest, SpatialFilterTest, PKIAuthenticationIntegrationTest, DistributedQueryComponentOptimizationTest, TestGraphTermsQParserPlugin, TestConfigSetsAPIZkFailure, TestLegacyTerms, TestRuleBasedAuthorizationPlugin, VersionInfoTest, TestCollectionAPI, ZkControllerTest, EnumFieldTest, ZkSolrClientTest, SolrCmdDistributorTest, TestCoreDiscovery, BadCopyFieldTest, JsonLoaderTest, ClassificationUpdateProcessorTest, BlockJoinFacetSimpleTest, DistribJoinFromCollectionTest, AddBlockUpdateTest, TestOnReconnectListenerSupport, CdcrVersionReplicationTest, TestUpdate, TestImplicitCoreProperties, TestMultiWordSynonyms, TestStressUserVersions, SegmentsInfoRequestHandlerTest, TolerantUpdateProcessorTest, ShardRoutingCustomTest, TestSolrConfigHandlerCloud, TestExportWriter, TestSearchPerf, UpdateRequestProcessorFactoryTest, TestJavabinTupleStreamParser, DirectSolrSpellCheckerTest, TestSchemaManager, StatelessScriptUpdateProcessorFactoryTest, SmileWriterTest, TestNestedDocsSort, HdfsChaosMonkeyNothingIsSafeTest, ZkCLITest, TestDistributedSearch, CdcrRequestHandlerTest, CloudExitableDirectoryReaderTest, CollectionReloadTest, CollectionTooManyReplicasTest, ConnectionManagerTest, CustomCollectionTest, DeleteShardTest, DistribDocExpirationUpdateProcessorTest, MoveReplicaHDFSTest, PeerSyncReplicationTest, RecoveryAfterSoftCommitTest, SaslZkACLProviderTest, SharedFSAutoReplicaFailoverUtilsTest, SimpleCollectionCreateDeleteTest, SolrCLIZkUtilsTest, SolrXmlInZkTest, TestAuthenticationFramework, TestCloudDeleteByQuery, TestCloudInspectUtil, TestCloudJSONFacetJoinDomain, TestCloudPivotFacet, TestCloudRecovery, TestClusterProperties, TestDeleteCollectionOnDownNodes, TestDistribDocBasedVersion, TestExclusionRuleCollectionAccess, TestHdfsCloudBackupRestore, TestLeaderInitiatedRecoveryThread, TestLocalFSCloudBackupRestore, TestMiniSolrCloudClusterSSL, TestPullReplicaErrorHandling, TestRequestStatusCollectionAPI, TestSolrCloudWithDelegationTokens, TestSolrCloudWithKerberosAlt, TestSolrCloudWithSecureImpersonation, TestStressInPlaceUpdates, HdfsBasicDistributedZkTest, HdfsThreadLeakTest, HdfsUnloadDistributedZkTest]
   [junit4] Completed [571/726 (1!)] on J1 in 59.20s, 1 test, 1 error <<< FAILURES!

[...truncated 43790 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 3 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/3/

2 tests failed.
FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
Didn't see all replicas for shard shard1 in c8n_1x2 come up within 90000 ms! ClusterState: {   "collMinRf_1x3":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{"shard1":{         "range":"80000000-7fffffff",         "state":"active",         "replicas":{           "core_node1":{             "core":"collMinRf_1x3_shard1_replica_t1",             "base_url":"http://127.0.0.1:44620",             "node_name":"127.0.0.1:44620_",             "state":"active",             "type":"TLOG"},           "core_node2":{             "core":"collMinRf_1x3_shard1_replica_t3",             "base_url":"http://127.0.0.1:50630",             "node_name":"127.0.0.1:50630_",             "state":"active",             "type":"TLOG"},           "core_node3":{             "core":"collMinRf_1x3_shard1_replica_t2",             "base_url":"http://127.0.0.1:46414",             "node_name":"127.0.0.1:46414_",             "state":"active",             "type":"TLOG",             "leader":"true"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"0",     "tlogReplicas":"3"},   "collection1":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{       "shard1":{         "range":"80000000-ffffffff",         "state":"active",         "replicas":{"core_node2":{             "core":"collection1_shard1_replica_t0",             "base_url":"http://127.0.0.1:50630",             "node_name":"127.0.0.1:50630_",             "state":"active",             "type":"TLOG",             "leader":"true"}}},       "shard2":{         "range":"0-7fffffff",         "state":"active",         "replicas":{           "core_node1":{             "core":"collection1_shard2_replica_t0",             "base_url":"http://127.0.0.1:38211",             "node_name":"127.0.0.1:38211_",             "state":"active",             "type":"TLOG",             "leader":"true"},           "core_node3":{             "core":"collection1_shard2_replica_t1",             "base_url":"http://127.0.0.1:46414",             "node_name":"127.0.0.1:46414_",             "state":"active",             "type":"TLOG"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"1",     "tlogReplicas":"0"},   "control_collection":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{"shard1":{         "range":"80000000-7fffffff",         "state":"active",         "replicas":{"core_node1":{             "core":"control_collection_shard1_replica_n1",             "base_url":"http://127.0.0.1:44620",             "node_name":"127.0.0.1:44620_",             "state":"active",             "type":"NRT",             "leader":"true"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"1",     "tlogReplicas":"0"},   "c8n_1x2":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{"shard1":{         "range":"80000000-7fffffff",         "state":"active",         "replicas":{           "core_node1":{             "state":"down",             "base_url":"http://127.0.0.1:46414",             "core":"c8n_1x2_shard1_replica_t1",             "node_name":"127.0.0.1:46414_",             "type":"TLOG"},           "core_node2":{             "core":"c8n_1x2_shard1_replica_t2",             "base_url":"http://127.0.0.1:38211",             "node_name":"127.0.0.1:38211_",             "state":"active",             "type":"TLOG",             "leader":"true"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"0",     "tlogReplicas":"2"}}

Stack Trace:
java.lang.AssertionError: Didn't see all replicas for shard shard1 in c8n_1x2 come up within 90000 ms! ClusterState: {
  "collMinRf_1x3":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{
          "core_node1":{
            "core":"collMinRf_1x3_shard1_replica_t1",
            "base_url":"http://127.0.0.1:44620",
            "node_name":"127.0.0.1:44620_",
            "state":"active",
            "type":"TLOG"},
          "core_node2":{
            "core":"collMinRf_1x3_shard1_replica_t3",
            "base_url":"http://127.0.0.1:50630",
            "node_name":"127.0.0.1:50630_",
            "state":"active",
            "type":"TLOG"},
          "core_node3":{
            "core":"collMinRf_1x3_shard1_replica_t2",
            "base_url":"http://127.0.0.1:46414",
            "node_name":"127.0.0.1:46414_",
            "state":"active",
            "type":"TLOG",
            "leader":"true"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"0",
    "tlogReplicas":"3"},
  "collection1":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{
      "shard1":{
        "range":"80000000-ffffffff",
        "state":"active",
        "replicas":{"core_node2":{
            "core":"collection1_shard1_replica_t0",
            "base_url":"http://127.0.0.1:50630",
            "node_name":"127.0.0.1:50630_",
            "state":"active",
            "type":"TLOG",
            "leader":"true"}}},
      "shard2":{
        "range":"0-7fffffff",
        "state":"active",
        "replicas":{
          "core_node1":{
            "core":"collection1_shard2_replica_t0",
            "base_url":"http://127.0.0.1:38211",
            "node_name":"127.0.0.1:38211_",
            "state":"active",
            "type":"TLOG",
            "leader":"true"},
          "core_node3":{
            "core":"collection1_shard2_replica_t1",
            "base_url":"http://127.0.0.1:46414",
            "node_name":"127.0.0.1:46414_",
            "state":"active",
            "type":"TLOG"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"1",
    "tlogReplicas":"0"},
  "control_collection":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{"core_node1":{
            "core":"control_collection_shard1_replica_n1",
            "base_url":"http://127.0.0.1:44620",
            "node_name":"127.0.0.1:44620_",
            "state":"active",
            "type":"NRT",
            "leader":"true"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"1",
    "tlogReplicas":"0"},
  "c8n_1x2":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{
          "core_node1":{
            "state":"down",
            "base_url":"http://127.0.0.1:46414",
            "core":"c8n_1x2_shard1_replica_t1",
            "node_name":"127.0.0.1:46414_",
            "type":"TLOG"},
          "core_node2":{
            "core":"c8n_1x2_shard1_replica_t2",
            "base_url":"http://127.0.0.1:38211",
            "node_name":"127.0.0.1:38211_",
            "state":"active",
            "type":"TLOG",
            "leader":"true"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"0",
    "tlogReplicas":"2"}}
        at __randomizedtesting.SeedInfo.seed([7A8BA450BDCA7B18:F2DF9B8A133616E0]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.ensureAllReplicasAreActive(AbstractFullDistribZkTestBase.java:1992)
        at org.apache.solr.cloud.HttpPartitionTest.testRf2(HttpPartitionTest.java:367)
        at org.apache.solr.cloud.HttpPartitionTest.test(HttpPartitionTest.java:131)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig

Error Message:
The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.

Stack Trace:
java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
        at __randomizedtesting.SeedInfo.seed([7A8BA450BDCA7B18:8D246D7B40439133]:0)
        at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:311)
        at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:287)
        at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:227)
        at org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig(HdfsDirectoryFactoryTest.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:693)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
        at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:71)
        at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:309)
        ... 42 more




Build Log:
[...truncated 11642 lines...]
   [junit4] Suite: org.apache.solr.core.HdfsDirectoryFactoryTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_7A8BA450BDCA7B18-001/init-core-data-001
   [junit4]   2> 750020 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1200 numCloses=1200
   [junit4]   2> 750021 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 750022 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 750069 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 750089 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 750099 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 750118 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_39176_hdfs____.l4ze0c/webapp
   [junit4]   2> 750646 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39176
   [junit4]   2> 751058 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 751060 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 751073 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_34466_datanode____.mke3l2/webapp
   [junit4]   2> 751540 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34466
   [junit4]   2> 751589 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 751591 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 751611 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_55062_datanode____4gpzqr/webapp
   [junit4]   2> 751751 INFO  (IPC Server handler 3 on 58243) [    ] BlockStateChange BLOCK* processReport: from storage DS-7710f540-795f-4161-abb8-4c08b3f2a2f3 node DatanodeRegistration(127.0.0.1:55148, datanodeUuid=fc0ef707-cff7-4adb-94f0-2b2f6512f449, infoPort=38625, infoSecurePort=0, ipcPort=43747, storageInfo=lv=-56;cid=testClusterID;nsid=1630742635;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 751751 INFO  (IPC Server handler 3 on 58243) [    ] BlockStateChange BLOCK* processReport: from storage DS-1bd35e09-7fbe-4ae6-b44b-6c65941b2f79 node DatanodeRegistration(127.0.0.1:55148, datanodeUuid=fc0ef707-cff7-4adb-94f0-2b2f6512f449, infoPort=38625, infoSecurePort=0, ipcPort=43747, storageInfo=lv=-56;cid=testClusterID;nsid=1630742635;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 752135 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:55062
   [junit4]   2> 752273 INFO  (IPC Server handler 5 on 58243) [    ] BlockStateChange BLOCK* processReport: from storage DS-da0bfe5a-2d46-4aac-8d57-ca1fdeb611b2 node DatanodeRegistration(127.0.0.1:57353, datanodeUuid=3fd4f5ea-1740-41e1-95e3-c68250a73fa7, infoPort=55137, infoSecurePort=0, ipcPort=43347, storageInfo=lv=-56;cid=testClusterID;nsid=1630742635;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 752273 INFO  (IPC Server handler 5 on 58243) [    ] BlockStateChange BLOCK* processReport: from storage DS-f50923a8-0794-4e52-bb01-20c69cea9925 node DatanodeRegistration(127.0.0.1:57353, datanodeUuid=3fd4f5ea-1740-41e1-95e3-c68250a73fa7, infoPort=55137, infoSecurePort=0, ipcPort=43347, storageInfo=lv=-56;cid=testClusterID;nsid=1630742635;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 752418 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testInitArgsOrSysPropConfig
   [junit4]   2> 752418 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:58243/solr1
   [junit4]   2> 752418 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 752420 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:58243/solr2
   [junit4]   2> 752420 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 752420 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:58243/solr2
   [junit4]   2> 752420 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 752420 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58243/solr
   [junit4]   2> 752428 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 752428 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [134217728] will allocate [1] slabs and use ~[134217728] bytes
   [junit4]   2> 752428 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 753533 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testInitArgsOrSysPropConfig
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HdfsDirectoryFactoryTest -Dtests.method=testInitArgsOrSysPropConfig -Dtests.seed=7A8BA450BDCA7B18 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=hr -Dtests.timezone=Pacific/Pago_Pago -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   1.16s J0 | HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
   [junit4]    > at __randomizedtesting.SeedInfo.seed([7A8BA450BDCA7B18:8D246D7B40439133]:0)
   [junit4]    > at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:311)
   [junit4]    > at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:287)
   [junit4]    > at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:227)
   [junit4]    > at org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig(HdfsDirectoryFactoryTest.java:111)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Direct buffer memory
   [junit4]    > at java.nio.Bits.reserveMemory(Bits.java:693)
   [junit4]    > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
   [junit4]    > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
   [junit4]    > at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:71)
   [junit4]    > at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:309)
   [junit4]    > ... 42 more
   [junit4]   2> 753581 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testLocalityReporter
   [junit4]   2> 753589 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:58243/solr
   [junit4]   2> 753589 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 753590 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58243/solr3/
   [junit4]   2> 753604 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.h.HdfsLocalityReporter Registering direcotry hdfs://localhost:58243/solr3 for locality metrics.
   [junit4]   2> 753630 INFO  (IPC Server handler 5 on 58243) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:55148 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f50923a8-0794-4e52-bb01-20c69cea9925:NORMAL:127.0.0.1:57353|RBW], ReplicaUC[[DISK]DS-1bd35e09-7fbe-4ae6-b44b-6c65941b2f79:NORMAL:127.0.0.1:55148|FINALIZED]]} size 0
   [junit4]   2> 753634 INFO  (IPC Server handler 7 on 58243) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57353 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f50923a8-0794-4e52-bb01-20c69cea9925:NORMAL:127.0.0.1:57353|RBW], ReplicaUC[[DISK]DS-1bd35e09-7fbe-4ae6-b44b-6c65941b2f79:NORMAL:127.0.0.1:55148|FINALIZED]]} size 0
   [junit4]   2> 753643 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testLocalityReporter
   [junit4]   2> 753681 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCleanupOldIndexDirectories
   [junit4]   2> 753681 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:58243/solr1
   [junit4]   2> 753681 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 753722 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.HdfsDirectoryFactory Found 1 old index directories to clean-up under hdfs://localhost:58243/solr1/mock/data afterReload=false
   [junit4]   2> 753723 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.HdfsDirectoryFactory Deleted old index directory hdfs://localhost:58243/solr1/mock/data/index.20170706162124380
   [junit4]   2> 753724 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCleanupOldIndexDirectories
   [junit4]   2> 753728 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 753750 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 753852 WARN  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_7A8BA450BDCA7B18-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_7A8BA450BDCA7B18-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to localhost/127.0.0.1:58243) [    ] o.a.h.h.s.d.DataNode BPOfferService for Block pool BP-1564204768-10.41.0.5-1499397680733 (Datanode Uuid 3fd4f5ea-1740-41e1-95e3-c68250a73fa7) service to localhost/127.0.0.1:58243 interrupted
   [junit4]   2> 753852 WARN  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_7A8BA450BDCA7B18-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_7A8BA450BDCA7B18-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to localhost/127.0.0.1:58243) [    ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1564204768-10.41.0.5-1499397680733 (Datanode Uuid 3fd4f5ea-1740-41e1-95e3-c68250a73fa7) service to localhost/127.0.0.1:58243
   [junit4]   2> 753858 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 753868 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 753969 WARN  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_7A8BA450BDCA7B18-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_7A8BA450BDCA7B18-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to localhost/127.0.0.1:58243) [    ] o.a.h.h.s.d.DataNode BPOfferService for Block pool BP-1564204768-10.41.0.5-1499397680733 (Datanode Uuid fc0ef707-cff7-4adb-94f0-2b2f6512f449) service to localhost/127.0.0.1:58243 interrupted
   [junit4]   2> 753971 WARN  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_7A8BA450BDCA7B18-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_7A8BA450BDCA7B18-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to localhost/127.0.0.1:58243) [    ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1564204768-10.41.0.5-1499397680733 (Datanode Uuid fc0ef707-cff7-4adb-94f0-2b2f6512f449) service to localhost/127.0.0.1:58243
   [junit4]   2> 753986 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 753987 WARN  (353659743@qtp-842560817-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39176) [    ] o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is false. Rechecking.
   [junit4]   2> 753987 WARN  (353659743@qtp-842560817-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39176) [    ] o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is false
   [junit4]   2> 754096 ERROR (SUITE-HdfsDirectoryFactoryTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.a.h.m.l.MethodMetric Error invoking method getBlocksTotal
   [junit4]   2> java.lang.reflect.InvocationTargetException
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MethodMetric$2.snapshot(MethodMetric.java:111)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MethodMetric.snapshot(MethodMetric.java:144)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MetricsRegistry.snapshot(MetricsRegistry.java:401)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder$1.getMetrics(MetricsSourceBuilder.java:79)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:194)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1804)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1595)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2> at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2> at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:226)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:204)
   [junit4]   2> at org.apache.solr.core.HdfsDirectoryFactoryTest.teardownClass(HdfsDirectoryFactoryTest.java:64)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> ... 54 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_7A8BA450BDCA7B18-001
   [junit4]   2> Jul 07, 2017 3:21:24 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 34 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=false): {}, locale=hr, timezone=Pacific/Pago_Pago
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=2,free=293625984,total=438304768
   [junit4]   2> NOTE: All tests run in this JVM: [TestManagedSchemaThreadSafety, TestMergePolicyConfig, TestConfigSets, TestNumericRangeQuery32, TestReqParamsAPI, TestSerializedLuceneMatchVersion, PeerSyncWithIndexFingerprintCachingTest, ChaosMonkeySafeLeaderWithPullReplicasTest, DistribDocExpirationUpdateProcessorTest, SolrCoreTest, ClassificationUpdateProcessorFactoryTest, TestCloudDeleteByQuery, TemplateUpdateProcessorTest, TestJoin, TestOrdValues, TestRangeQuery, TestRemoteStreaming, HighlighterConfigTest, RuleEngineTest, TestSha256AuthenticationProvider, TestBinaryResponseWriter, SolrIndexSplitterTest, SolrIndexConfigTest, CdcrBootstrapTest, BinaryUpdateRequestHandlerTest, FieldMutatingUpdateProcessorTest, TestScoreJoinQPNoScore, DeleteLastCustomShardedReplicaTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestSolrCloudWithHadoopAuthPlugin, HdfsBasicDistributedZkTest, HdfsDirectoryFactoryTest]
   [junit4] Completed [113/727 (1!)] on J0 in 13.81s, 3 tests, 1 error <<< FAILURES!

[...truncated 1374 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/init-core-data-001
   [junit4]   2> 3314555 INFO  (SUITE-HttpPartitionTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 3314556 INFO  (SUITE-HttpPartitionTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 3314557 INFO  (SUITE-HttpPartitionTest-seed#[7A8BA450BDCA7B18]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 3314561 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3314562 INFO  (Thread-134830) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3314562 INFO  (Thread-134830) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3314566 ERROR (Thread-134830) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3314662 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.ZkTestServer start zk server on port:51766
   [junit4]   2> 3314680 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3314683 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 3314685 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3314688 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3314690 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3314692 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3314693 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3314695 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3314698 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3314719 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3314725 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3314726 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15d1b38100a0001, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 3314726 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use TLOG replicas unless explicitly asked otherwise
   [junit4]   2> 3314882 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3314884 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6d78fb8{/,null,AVAILABLE}
   [junit4]   2> 3314888 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4d6f690a{HTTP/1.1,[http/1.1]}{127.0.0.1:60359}
   [junit4]   2> 3314889 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.e.j.s.Server Started @3317967ms
   [junit4]   2> 3314889 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=44620, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/control-001/cores}
   [junit4]   2> 3314889 ERROR (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3314889 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 3314889 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3314889 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 3314890 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-07T04:04:05.584Z
   [junit4]   2> 3314893 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3314893 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/control-001/solr.xml
   [junit4]   2> 3314899 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@2cb5351d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3314902 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51766/solr
   [junit4]   2> 3315019 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:44620_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3315021 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:44620_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:44620_
   [junit4]   2> 3315026 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:44620_    ] o.a.s.c.Overseer Overseer (id=98264694473228292-127.0.0.1:44620_-n_0000000000) starting
   [junit4]   2> 3315036 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:44620_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44620_
   [junit4]   2> 3315037 INFO  (zkCallback-2281-thread-1-processing-n:127.0.0.1:44620_) [n:127.0.0.1:44620_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3315137 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:44620_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cb5351d
   [junit4]   2> 3315143 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:44620_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cb5351d
   [junit4]   2> 3315144 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:44620_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cb5351d
   [junit4]   2> 3315145 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:44620_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/control-001/cores
   [junit4]   2> 3315168 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3315169 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:51766/solr ready
   [junit4]   2> 3315170 INFO  (SocketProxy-Acceptor-44620) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=34823,localport=44620], receiveBufferSize:531000
   [junit4]   2> 3315174 INFO  (SocketProxy-Acceptor-44620) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=60359,localport=49364], receiveBufferSize=530904
   [junit4]   2> 3315176 INFO  (qtp1020669270-150232) [n:127.0.0.1:44620_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:44620_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3315179 INFO  (OverseerThreadFactory-7253-thread-1-processing-n:127.0.0.1:44620_) [n:127.0.0.1:44620_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 3315284 INFO  (SocketProxy-Acceptor-44620) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=34825,localport=44620], receiveBufferSize:531000
   [junit4]   2> 3315284 INFO  (SocketProxy-Acceptor-44620) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=60359,localport=49366], receiveBufferSize=530904
   [junit4]   2> 3315286 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3315288 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3315392 INFO  (zkCallback-2281-thread-1-processing-n:127.0.0.1:44620_) [n:127.0.0.1:44620_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3316341 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3316364 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 3316503 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3316521 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 3316521 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cb5351d
   [junit4]   2> 3316521 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3316521 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 3316525 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=877205642, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 3316528 WARN  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3316592 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3316592 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3316593 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3316593 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3316594 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@1618b770
   [junit4]   2> 3316595 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@10457e2e[control_collection_shard1_replica_n1] main]
   [junit4]   2> 3316598 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3316598 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3316599 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3316601 INFO  (searcherExecutor-7256-thread-1-processing-n:127.0.0.1:44620_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@10457e2e[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3316601 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572235113707601920
   [junit4]   2> 3316613 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3316613 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3316613 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:44620/control_collection_shard1_replica_n1/
   [junit4]   2> 3316613 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3316613 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:44620/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 3316613 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3316618 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:44620/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 3316722 INFO  (zkCallback-2281-thread-1-processing-n:127.0.0.1:44620_) [n:127.0.0.1:44620_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3316771 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3316773 INFO  (qtp1020669270-150234) [n:127.0.0.1:44620_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1486
   [junit4]   2> 3316775 INFO  (qtp1020669270-150232) [n:127.0.0.1:44620_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 3316875 INFO  (zkCallback-2281-thread-1-processing-n:127.0.0.1:44620_) [n:127.0.0.1:44620_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3317180 INFO  (OverseerCollectionConfigSetProcessor-98264694473228292-127.0.0.1:44620_-n_0000000000) [n:127.0.0.1:44620_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3317775 INFO  (qtp1020669270-150232) [n:127.0.0.1:44620_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:44620_&wt=javabin&version=2} status=0 QTime=2599
   [junit4]   2> 3317787 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3317788 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:51766/solr ready
   [junit4]   2> 3317788 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3317788 INFO  (SocketProxy-Acceptor-44620) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=34854,localport=44620], receiveBufferSize:531000
   [junit4]   2> 3317789 INFO  (SocketProxy-Acceptor-44620) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=60359,localport=49395], receiveBufferSize=530904
   [junit4]   2> 3317792 INFO  (qtp1020669270-150230) [n:127.0.0.1:44620_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3317794 INFO  (OverseerThreadFactory-7253-thread-2-processing-n:127.0.0.1:44620_) [n:127.0.0.1:44620_    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 3317794 WARN  (OverseerThreadFactory-7253-thread-2-processing-n:127.0.0.1:44620_) [n:127.0.0.1:44620_    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 3317999 INFO  (qtp1020669270-150230) [n:127.0.0.1:44620_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 3318000 INFO  (qtp1020669270-150230) [n:127.0.0.1:44620_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=208
   [junit4]   2> 3318122 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/shard-1-001 of type TLOG
   [junit4]   2> 3318123 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3318124 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6f5ac709{/,null,AVAILABLE}
   [junit4]   2> 3318125 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@28da5dc4{HTTP/1.1,[http/1.1]}{127.0.0.1:37897}
   [junit4]   2> 3318126 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.e.j.s.Server Started @3321204ms
   [junit4]   2> 3318126 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/tempDir-001/jetty1, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=38211, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/shard-1-001/cores}
   [junit4]   2> 3318127 ERROR (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3318127 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 3318127 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3318127 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 3318127 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-07T04:04:08.821Z
   [junit4]   2> 3318130 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3318130 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/shard-1-001/solr.xml
   [junit4]   2> 3318135 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@2cb5351d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3318138 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51766/solr
   [junit4]   2> 3318148 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:38211_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3318150 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:38211_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3318152 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:38211_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38211_
   [junit4]   2> 3318153 INFO  (zkCallback-2281-thread-1-processing-n:127.0.0.1:44620_) [n:127.0.0.1:44620_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3318154 INFO  (zkCallback-2288-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3318156 INFO  (zkCallback-2293-thread-1-processing-n:127.0.0.1:38211_) [n:127.0.0.1:38211_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3318309 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:38211_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cb5351d
   [junit4]   2> 3318315 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:38211_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cb5351d
   [junit4]   2> 3318315 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:38211_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cb5351d
   [junit4]   2> 3318316 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:38211_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/shard-1-001/cores
   [junit4]   2> 3318341 INFO  (SocketProxy-Acceptor-38211) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=51826,localport=38211], receiveBufferSize:531000
   [junit4]   2> 3318344 INFO  (SocketProxy-Acceptor-38211) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=37897,localport=37731], receiveBufferSize=530904
   [junit4]   2> 3318347 INFO  (qtp121657135-150280) [n:127.0.0.1:38211_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:38211_&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3318349 INFO  (OverseerCollectionConfigSetProcessor-98264694473228292-127.0.0.1:44620_-n_0000000000) [n:127.0.0.1:44620_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3318350 INFO  (OverseerThreadFactory-7253-thread-3-processing-n:127.0.0.1:44620_) [n:127.0.0.1:44620_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:38211_ for creating new replica
   [junit4]   2> 3318351 INFO  (SocketProxy-Acceptor-38211) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=51828,localport=38211], receiveBufferSize:531000
   [junit4]   2> 3318351 INFO  (SocketProxy-Acceptor-38211) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=37897,localport=37733], receiveBufferSize=530904
   [junit4]   2> 3318353 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 3318353 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3318458 INFO  (zkCallback-2293-thread-1-processing-n:127.0.0.1:38211_) [n:127.0.0.1:38211_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3319369 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3319385 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.s.IndexSchema [collection1_shard2_replica_t0] Schema name=test
   [junit4]   2> 3319519 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3319535 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_t0' using configuration from collection collection1, trusted=true
   [junit4]   2> 3319535 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_t0' (registry 'solr.core.collection1.shard2.replica_t0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cb5351d
   [junit4]   2> 3319536 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3319536 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.SolrCore [[collection1_shard2_replica_t0] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/shard-1-001/cores/collection1_shard2_replica_t0], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/shard-1-001/cores/collection1_shard2_replica_t0/data/]
   [junit4]   2> 3319538 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=877205642, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 3319541 WARN  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3319597 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3319597 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3319599 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3319599 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3319600 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@2279203e
   [junit4]   2> 3319600 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.s.SolrIndexSearcher Opening [Searcher@5f3eb561[collection1_shard2_replica_t0] main]
   [junit4]   2> 3319601 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3319602 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3319602 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3319604 INFO  (searcherExecutor-7267-thread-1-processing-n:127.0.0.1:38211_ x:collection1_shard2_replica_t0 s:shard2 c:collection1) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.SolrCore [collection1_shard2_replica_t0] Registered new searcher Searcher@5f3eb561[collection1_shard2_replica_t0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3319604 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572235116856475648
   [junit4]   2> 3319610 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3319610 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3319610 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:38211/collection1_shard2_replica_t0/
   [junit4]   2> 3319610 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3319610 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.SyncStrategy http://127.0.0.1:38211/collection1_shard2_replica_t0/ has no replicas
   [junit4]   2> 3319610 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3319611 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.ZkController collection1_shard2_replica_t0 stopping background replication from leader
   [junit4]   2> 3319614 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:38211/collection1_shard2_replica_t0/ shard2
   [junit4]   2> 3319716 INFO  (zkCallback-2293-thread-1-processing-n:127.0.0.1:38211_) [n:127.0.0.1:38211_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3319765 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3319767 INFO  (qtp121657135-150282) [n:127.0.0.1:38211_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1414
   [junit4]   2> 3319770 INFO  (qtp121657135-150280) [n:127.0.0.1:38211_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:38211_&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2} status=0 QTime=1423
   [junit4]   2> 3319869 INFO  (zkCallback-2293-thread-1-processing-n:127.0.0.1:38211_) [n:127.0.0.1:38211_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3319895 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/shard-2-001 of type TLOG
   [junit4]   2> 3319895 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3319897 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@65396d44{/,null,AVAILABLE}
   [junit4]   2> 3319897 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1611a288{HTTP/1.1,[http/1.1]}{127.0.0.1:43438}
   [junit4]   2> 3319899 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.e.j.s.Server Started @3322977ms
   [junit4]   2> 3319899 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/tempDir-001/jetty2, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=50630, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/shard-2-001/cores}
   [junit4]   2> 3319899 ERROR (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3319900 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 3319900 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3319900 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 3319900 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-07T04:04:10.594Z
   [junit4]   2> 3319903 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3319903 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001/shard-2-001/solr.xml
   [junit4]   2> 3319908 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@2cb5351d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3319911 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51766/solr
   [junit4]   2> 3319920 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:50630_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3319923 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:50630_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3319925 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:50630_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50630_
   [junit4]   2> 3319926 INFO  (zkCallback-2293-thread-1-processing-n:127.0.0.1:38211_) [n:127.0.0.1:38211_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3319926 INFO  (zkCallback-2288-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3319926 INFO  (zkCallback-2281-thread-1-processing-n:127.0.0.1:44620_) [n:127.0.0.1:44620_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3319928 INFO  (zkCallback-2299-thread-1-processing-n:127.0.0.1:50630_) [n:127.0.0.1:50630_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3320019 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:50630_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cb5351d
   [junit4]   2> 3320024 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:50630_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cb5351d
   [junit4]   2> 3320025 INFO  (TEST-HttpPartitionTest.test-seed#[7A8BA450BDCA7B18]) [n:127.0.0.1:50630_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cb5351d
   [

[...truncated too long message...]

":{
   [junit4]    >       "shard1":{
   [junit4]    >         "range":"80000000-ffffffff",
   [junit4]    >         "state":"active",
   [junit4]    >         "replicas":{"core_node2":{
   [junit4]    >             "core":"collection1_shard1_replica_t0",
   [junit4]    >             "base_url":"http://127.0.0.1:50630",
   [junit4]    >             "node_name":"127.0.0.1:50630_",
   [junit4]    >             "state":"active",
   [junit4]    >             "type":"TLOG",
   [junit4]    >             "leader":"true"}}},
   [junit4]    >       "shard2":{
   [junit4]    >         "range":"0-7fffffff",
   [junit4]    >         "state":"active",
   [junit4]    >         "replicas":{
   [junit4]    >           "core_node1":{
   [junit4]    >             "core":"collection1_shard2_replica_t0",
   [junit4]    >             "base_url":"http://127.0.0.1:38211",
   [junit4]    >             "node_name":"127.0.0.1:38211_",
   [junit4]    >             "state":"active",
   [junit4]    >             "type":"TLOG",
   [junit4]    >             "leader":"true"},
   [junit4]    >           "core_node3":{
   [junit4]    >             "core":"collection1_shard2_replica_t1",
   [junit4]    >             "base_url":"http://127.0.0.1:46414",
   [junit4]    >             "node_name":"127.0.0.1:46414_",
   [junit4]    >             "state":"active",
   [junit4]    >             "type":"TLOG"}}}},
   [junit4]    >     "router":{"name":"compositeId"},
   [junit4]    >     "maxShardsPerNode":"1",
   [junit4]    >     "autoAddReplicas":"false",
   [junit4]    >     "nrtReplicas":"1",
   [junit4]    >     "tlogReplicas":"0"},
   [junit4]    >   "control_collection":{
   [junit4]    >     "pullReplicas":"0",
   [junit4]    >     "replicationFactor":"1",
   [junit4]    >     "shards":{"shard1":{
   [junit4]    >         "range":"80000000-7fffffff",
   [junit4]    >         "state":"active",
   [junit4]    >         "replicas":{"core_node1":{
   [junit4]    >             "core":"control_collection_shard1_replica_n1",
   [junit4]    >             "base_url":"http://127.0.0.1:44620",
   [junit4]    >             "node_name":"127.0.0.1:44620_",
   [junit4]    >             "state":"active",
   [junit4]    >             "type":"NRT",
   [junit4]    >             "leader":"true"}}}},
   [junit4]    >     "router":{"name":"compositeId"},
   [junit4]    >     "maxShardsPerNode":"1",
   [junit4]    >     "autoAddReplicas":"false",
   [junit4]    >     "nrtReplicas":"1",
   [junit4]    >     "tlogReplicas":"0"},
   [junit4]    >   "c8n_1x2":{
   [junit4]    >     "pullReplicas":"0",
   [junit4]    >     "replicationFactor":"1",
   [junit4]    >     "shards":{"shard1":{
   [junit4]    >         "range":"80000000-7fffffff",
   [junit4]    >         "state":"active",
   [junit4]    >         "replicas":{
   [junit4]    >           "core_node1":{
   [junit4]    >             "state":"down",
   [junit4]    >             "base_url":"http://127.0.0.1:46414",
   [junit4]    >             "core":"c8n_1x2_shard1_replica_t1",
   [junit4]    >             "node_name":"127.0.0.1:46414_",
   [junit4]    >             "type":"TLOG"},
   [junit4]    >           "core_node2":{
   [junit4]    >             "core":"c8n_1x2_shard1_replica_t2",
   [junit4]    >             "base_url":"http://127.0.0.1:38211",
   [junit4]    >             "node_name":"127.0.0.1:38211_",
   [junit4]    >             "state":"active",
   [junit4]    >             "type":"TLOG",
   [junit4]    >             "leader":"true"}}}},
   [junit4]    >     "router":{"name":"compositeId"},
   [junit4]    >     "maxShardsPerNode":"1",
   [junit4]    >     "autoAddReplicas":"false",
   [junit4]    >     "nrtReplicas":"0",
   [junit4]    >     "tlogReplicas":"2"}}
   [junit4]    > at __randomizedtesting.SeedInfo.seed([7A8BA450BDCA7B18:F2DF9B8A133616E0]:0)
   [junit4]    > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.ensureAllReplicasAreActive(AbstractFullDistribZkTestBase.java:1992)
   [junit4]    > at org.apache.solr.cloud.HttpPartitionTest.testRf2(HttpPartitionTest.java:367)
   [junit4]    > at org.apache.solr.cloud.HttpPartitionTest.test(HttpPartitionTest.java:131)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_7A8BA450BDCA7B18-001
   [junit4]   2> Jul 07, 2017 4:06:59 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {multiDefault=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a_t=FSTOrd50, id=PostingsFormat(name=LuceneVarGapFixedInterval), text=PostingsFormat(name=LuceneFixedGap)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Lucene70), _version_=DocValuesFormat(name=Direct), range_facet_i_dv=DocValuesFormat(name=Direct), intDvoDefault=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=1018, maxMBSortInHeap=5.550023862200683, sim=RandomSimilarity(queryNorm=false): {}, locale=es-UY, timezone=Africa/Lusaka
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=203124088,total=523763712
   [junit4]   2> NOTE: All tests run in this JVM: [TestManagedSchemaThreadSafety, TestMergePolicyConfig, TestConfigSets, TestNumericRangeQuery32, TestReqParamsAPI, TestSerializedLuceneMatchVersion, PeerSyncWithIndexFingerprintCachingTest, ChaosMonkeySafeLeaderWithPullReplicasTest, DistribDocExpirationUpdateProcessorTest, SolrCoreTest, ClassificationUpdateProcessorFactoryTest, TestCloudDeleteByQuery, TemplateUpdateProcessorTest, TestJoin, TestOrdValues, TestRangeQuery, TestRemoteStreaming, HighlighterConfigTest, RuleEngineTest, TestSha256AuthenticationProvider, TestBinaryResponseWriter, SolrIndexSplitterTest, SolrIndexConfigTest, CdcrBootstrapTest, BinaryUpdateRequestHandlerTest, FieldMutatingUpdateProcessorTest, TestScoreJoinQPNoScore, DeleteLastCustomShardedReplicaTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestSolrCloudWithHadoopAuthPlugin, HdfsBasicDistributedZkTest, HdfsDirectoryFactoryTest, TestTlogReplica, TestInPlaceUpdatesStandalone, DistributedFacetPivotSmallAdvancedTest, SpatialRPTFieldTypeTest, TestRealTimeGet, TestPHPSerializedResponseWriter, TestHighFrequencyDictionaryFactory, CoreSorterTest, BasicZkTest, TestWordDelimiterFilterFactory, TestSubQueryTransformerCrossCore, DebugComponentTest, ZkNodePropsTest, SuggestComponentTest, TestNoOpRegenerator, TestGeoJSONResponseWriter, AssignTest, TestDeleteCollectionOnDownNodes, DefaultValueUpdateProcessorTest, UpdateParamsTest, TestLockTree, TestHighlightDedupGrouping, TestStressLucene, WordBreakSolrSpellCheckerTest, BufferStoreTest, LargeFieldTest, TestBadConfig, RequestLoggingTest, SortSpecParsingTest, TestStressCloudBlindAtomicUpdates, TestFieldSortValues, MigrateRouteKeyTest, TestMacros, TestCollationField, ConnectionReuseTest, TestSchemaResource, TestLMDirichletSimilarityFactory, TestExtendedDismaxParser, TestManagedSynonymFilterFactory, TlogReplayBufferedWhileIndexingTest, TestMiniSolrCloudClusterSSL, TestQuerySenderNoQuery, ManagedSchemaRoundRobinCloudTest, TestCSVResponseWriter, DateRangeFieldTest, TestObjectReleaseTracker, TestConfigSetsAPIExclusivity, LeaderFailureAfterFreshStartTest, AutoScalingHandlerTest, TestIBSimilarityFactory, RestartWhileUpdatingTest, TestConfigOverlay, SecurityConfHandlerTest, CollectionTooManyReplicasTest, ExternalFileFieldSortTest, HLLUtilTest, TestSolrCloudWithDelegationTokens, DistributedVersionInfoTest, HdfsWriteToMultipleCollectionsTest, TestSmileRequest, TestHalfAndHalfDocValues, MoveReplicaTest, TestBinaryField, ShufflingReplicaListTransformerTest, TestClassNameShortening, DistributedQueryComponentCustomSortTest, ZkStateReaderTest, TestTrieFacet, CheckHdfsIndexTest, TestTestInjection, TestDocSet, TestExceedMaxTermLength, TestDynamicFieldResource, TestClusterProperties, DistributedQueryElevationComponentTest, OutputWriterTest, BasicAuthIntegrationTest, SharedFSAutoReplicaFailoverUtilsTest, TestCloudPivotFacet, TestLuceneMatchVersion, SolrRequestParserTest, TestLeaderElectionZkExpiry, TestSolrQueryParser, ChaosMonkeyNothingIsSafeTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, TestRandomFaceting, TestRandomDVFaceting, TestDistributedSearch, AutoCommitTest, TestFaceting, TestRecovery, ZkControllerTest, TestStressVersions, HardAutoCommitTest, SpellCheckComponentTest, QueryElevationComponentTest, PeerSyncTest, TestFunctionQuery, BasicFunctionalityTest, TestLazyCores, SoftAutoCommitTest, SortByFunctionTest, DocValuesMultiTest, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, DisMaxRequestHandlerTest, TestReversedWildcardFilterFactory, TestQueryTypes, FileBasedSpellCheckerTest, RequestHandlersTest, TestValueSourceCache, TermVectorComponentTest, RequiredFieldsTest, LoggingHandlerTest, IndexSchemaTest, UpdateRequestProcessorFactoryTest, TestAnalyzedSuggestions, TestComponentsName, SOLR749Test, CopyFieldTest, ResponseLogComponentTest, TestStressRecovery, DistributedIntervalFacetingTest, TestEmbeddedSolrServerConstructors, TestEmbeddedSolrServerSchemaAPI, ActionThrottleTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, ConcurrentDeleteAndCreateCollectionTest, DeleteInactiveReplicaTest, DeleteStatusTest, DistribJoinFromCollectionTest, DistributedQueueTest, HttpPartitionTest]
   [junit4] Completed [527/727 (2!)] on J0 in 174.90s, 1 test, 1 failure <<< FAILURES!

[...truncated 43993 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 4 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/4/

3 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([CAA5C5599C244612:42F1FA8332D82BEA]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test(ChaosMonkeySafeLeaderWithPullReplicasTest.java:174)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
9 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:     1) Thread[id=10416, name=StoppableCommitThread-SendThread(127.0.0.1:34126), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    2) Thread[id=8905, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    3) Thread[id=10417, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    4) Thread[id=10418, name=zkCallback-1016-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=8906, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    6) Thread[id=8907, name=StoppableCommitThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)         at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)         at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)         at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1124)         at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1110)         at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:646)         at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1227)         at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)         at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)         at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)         at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:49)    7) Thread[id=10455, name=zkCallback-1016-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    8) Thread[id=10457, name=zkCallback-1016-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    9) Thread[id=10456, name=zkCallback-1016-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 9 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:
   1) Thread[id=10416, name=StoppableCommitThread-SendThread(127.0.0.1:34126), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   2) Thread[id=8905, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   3) Thread[id=10417, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   4) Thread[id=10418, name=zkCallback-1016-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=8906, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   6) Thread[id=8907, name=StoppableCommitThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1124)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1110)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:646)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1227)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)
        at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:49)
   7) Thread[id=10455, name=zkCallback-1016-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=10457, name=zkCallback-1016-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   9) Thread[id=10456, name=zkCallback-1016-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([CAA5C5599C244612]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=10416, name=StoppableCommitThread-SendThread(127.0.0.1:34126), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    2) Thread[id=10418, name=zkCallback-1016-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=10455, name=zkCallback-1016-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=10457, name=zkCallback-1016-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=10456, name=zkCallback-1016-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=10416, name=StoppableCommitThread-SendThread(127.0.0.1:34126), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   2) Thread[id=10418, name=zkCallback-1016-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=10455, name=zkCallback-1016-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=10457, name=zkCallback-1016-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=10456, name=zkCallback-1016-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([CAA5C5599C244612]:0)




Build Log:
[...truncated 11847 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/init-core-data-001
   [junit4]   2> 712385 WARN  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[CAA5C5599C244612]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=58 numCloses=58
   [junit4]   2> 712386 INFO  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[CAA5C5599C244612]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 712387 INFO  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[CAA5C5599C244612]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 712387 INFO  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[CAA5C5599C244612]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 712387 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.ChaosMonkeySafeLeaderWithPullReplicasTest Starting ChaosMonkey test with 2 shards and 4 nodes
   [junit4]   2> 712392 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 712393 INFO  (Thread-3426) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 712393 INFO  (Thread-3426) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 712404 ERROR (Thread-3426) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 712493 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.ZkTestServer start zk server on port:34126
   [junit4]   2> 712504 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 712505 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 712506 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 712507 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 712509 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 712510 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 712511 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 712513 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 712514 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 712515 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 712516 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 712517 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use TLOG replicas unless explicitly asked otherwise
   [junit4]   2> 712603 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 712604 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3d21e1be{/,null,AVAILABLE}
   [junit4]   2> 712605 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1819b989{HTTP/1.1,[http/1.1]}{127.0.0.1:47631}
   [junit4]   2> 712605 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.e.j.s.Server Started @714794ms
   [junit4]   2> 712605 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/tempDir-001/control/data, hostContext=/, hostPort=47631, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/control-001/cores}
   [junit4]   2> 712606 ERROR (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 712606 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 712606 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 712606 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 712606 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-08T03:52:20.433Z
   [junit4]   2> 712615 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 712615 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/control-001/solr.xml
   [junit4]   2> 712619 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5cb7ef52, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 712621 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34126/solr
   [junit4]   2> 712702 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:47631_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 712703 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:47631_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:47631_
   [junit4]   2> 712704 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:47631_    ] o.a.s.c.Overseer Overseer (id=98270310578978820-127.0.0.1:47631_-n_0000000000) starting
   [junit4]   2> 712711 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:47631_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47631_
   [junit4]   2> 712712 INFO  (zkCallback-983-thread-1-processing-n:127.0.0.1:47631_) [n:127.0.0.1:47631_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 712800 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:47631_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5cb7ef52
   [junit4]   2> 712803 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:47631_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5cb7ef52
   [junit4]   2> 712803 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:47631_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5cb7ef52
   [junit4]   2> 712804 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:47631_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/control-001/cores
   [junit4]   2> 712817 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 712818 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34126/solr ready
   [junit4]   2> 712820 INFO  (qtp511552926-8737) [n:127.0.0.1:47631_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:47631_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 712822 INFO  (OverseerThreadFactory-2680-thread-1-processing-n:127.0.0.1:47631_) [n:127.0.0.1:47631_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 712928 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 712928 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 713032 INFO  (zkCallback-983-thread-1-processing-n:127.0.0.1:47631_) [n:127.0.0.1:47631_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 713942 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 713952 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 714021 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 714035 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 714036 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5cb7ef52
   [junit4]   2> 714036 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 714036 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 714038 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=10, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 714047 WARN  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 714078 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 714078 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 714078 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
   [junit4]   2> 714078 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 714079 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@5e6289bf
   [junit4]   2> 714080 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@701d8716[control_collection_shard1_replica_n1] main]
   [junit4]   2> 714081 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 714081 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 714082 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 714083 INFO  (searcherExecutor-2683-thread-1-processing-n:127.0.0.1:47631_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@701d8716[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 714083 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572324971024220160
   [junit4]   2> 714088 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 714088 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 714088 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:47631/control_collection_shard1_replica_n1/
   [junit4]   2> 714088 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 714088 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:47631/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 714088 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 714093 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:47631/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 714196 INFO  (zkCallback-983-thread-1-processing-n:127.0.0.1:47631_) [n:127.0.0.1:47631_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 714244 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 714246 INFO  (qtp511552926-8739) [n:127.0.0.1:47631_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1318
   [junit4]   2> 714249 INFO  (qtp511552926-8737) [n:127.0.0.1:47631_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 714350 INFO  (zkCallback-983-thread-2-processing-n:127.0.0.1:47631_) [n:127.0.0.1:47631_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 714823 INFO  (OverseerCollectionConfigSetProcessor-98270310578978820-127.0.0.1:47631_-n_0000000000) [n:127.0.0.1:47631_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 715249 INFO  (qtp511552926-8737) [n:127.0.0.1:47631_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:47631_&wt=javabin&version=2} status=0 QTime=2429
   [junit4]   2> 715253 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 715254 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34126/solr ready
   [junit4]   2> 715254 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 715255 INFO  (qtp511552926-8734) [n:127.0.0.1:47631_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 715256 INFO  (OverseerThreadFactory-2680-thread-2-processing-n:127.0.0.1:47631_) [n:127.0.0.1:47631_    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 715257 WARN  (OverseerThreadFactory-2680-thread-2-processing-n:127.0.0.1:47631_) [n:127.0.0.1:47631_    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 715463 INFO  (qtp511552926-8734) [n:127.0.0.1:47631_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 715463 INFO  (qtp511552926-8734) [n:127.0.0.1:47631_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=208
   [junit4]   2> 715550 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/shard-1-001 of type TLOG
   [junit4]   2> 715550 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 715552 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@55f2fe38{/,null,AVAILABLE}
   [junit4]   2> 715552 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@17edcbd{HTTP/1.1,[http/1.1]}{127.0.0.1:56662}
   [junit4]   2> 715552 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.e.j.s.Server Started @717741ms
   [junit4]   2> 715552 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/tempDir-001/jetty1, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=56662, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/shard-1-001/cores}
   [junit4]   2> 715552 ERROR (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 715553 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 715553 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 715553 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 715553 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-08T03:52:23.380Z
   [junit4]   2> 715556 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 715556 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/shard-1-001/solr.xml
   [junit4]   2> 715560 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5cb7ef52, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 715562 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34126/solr
   [junit4]   2> 715573 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:56662_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 715575 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:56662_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 715577 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:56662_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56662_
   [junit4]   2> 715579 INFO  (zkCallback-990-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 715579 INFO  (zkCallback-983-thread-2-processing-n:127.0.0.1:47631_) [n:127.0.0.1:47631_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 715580 INFO  (zkCallback-995-thread-1-processing-n:127.0.0.1:56662_) [n:127.0.0.1:56662_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 715658 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:56662_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5cb7ef52
   [junit4]   2> 715662 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:56662_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5cb7ef52
   [junit4]   2> 715663 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:56662_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5cb7ef52
   [junit4]   2> 715664 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:56662_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/shard-1-001/cores
   [junit4]   2> 715682 INFO  (qtp511552926-8735) [n:127.0.0.1:47631_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:56662_&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 715684 INFO  (OverseerCollectionConfigSetProcessor-98270310578978820-127.0.0.1:47631_-n_0000000000) [n:127.0.0.1:47631_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 715684 INFO  (OverseerThreadFactory-2680-thread-3-processing-n:127.0.0.1:47631_) [n:127.0.0.1:47631_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:56662_ for creating new replica
   [junit4]   2> 715686 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 715686 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 715789 INFO  (zkCallback-995-thread-1-processing-n:127.0.0.1:56662_) [n:127.0.0.1:56662_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 716699 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 716709 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.s.IndexSchema [collection1_shard2_replica_t0] Schema name=test
   [junit4]   2> 717402 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 717416 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_t0' using configuration from collection collection1, trusted=true
   [junit4]   2> 717417 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_t0' (registry 'solr.core.collection1.shard2.replica_t0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5cb7ef52
   [junit4]   2> 717417 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 717417 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.SolrCore [[collection1_shard2_replica_t0] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/shard-1-001/cores/collection1_shard2_replica_t0], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/shard-1-001/cores/collection1_shard2_replica_t0/data/]
   [junit4]   2> 717420 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=10, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 717428 WARN  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 717459 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 717460 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 717460 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
   [junit4]   2> 717460 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 717461 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@1a547132
   [junit4]   2> 717462 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.s.SolrIndexSearcher Opening [Searcher@619c1d5d[collection1_shard2_replica_t0] main]
   [junit4]   2> 717463 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 717463 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 717464 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 717464 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572324974569455616
   [junit4]   2> 717470 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 717470 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 717470 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:56662/collection1_shard2_replica_t0/
   [junit4]   2> 717470 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 717470 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.SyncStrategy http://127.0.0.1:56662/collection1_shard2_replica_t0/ has no replicas
   [junit4]   2> 717470 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 717471 INFO  (searcherExecutor-2694-thread-1-processing-n:127.0.0.1:56662_ x:collection1_shard2_replica_t0 s:shard2 c:collection1) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.SolrCore [collection1_shard2_replica_t0] Registered new searcher Searcher@619c1d5d[collection1_shard2_replica_t0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 717471 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.ZkController collection1_shard2_replica_t0 stopping background replication from leader
   [junit4]   2> 717474 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:56662/collection1_shard2_replica_t0/ shard2
   [junit4]   2> 717575 INFO  (zkCallback-995-thread-1-processing-n:127.0.0.1:56662_) [n:127.0.0.1:56662_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 717625 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 717626 INFO  (qtp377920575-8779) [n:127.0.0.1:56662_ c:collection1 s:shard2  x:collection1_shard2_replica_t0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1940
   [junit4]   2> 717628 INFO  (qtp511552926-8735) [n:127.0.0.1:47631_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:56662_&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2} status=0 QTime=1946
   [junit4]   2> 717690 INFO  (OverseerCollectionConfigSetProcessor-98270310578978820-127.0.0.1:47631_-n_0000000000) [n:127.0.0.1:47631_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 717713 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/shard-2-001 of type TLOG
   [junit4]   2> 717715 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 717717 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@42d50801{/,null,AVAILABLE}
   [junit4]   2> 717718 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@61b591b1{HTTP/1.1,[http/1.1]}{127.0.0.1:55556}
   [junit4]   2> 717718 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.e.j.s.Server Started @719907ms
   [junit4]   2> 717718 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/tempDir-001/jetty2, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=55556, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/shard-2-001/cores}
   [junit4]   2> 717718 ERROR (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 717726 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 717726 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 717726 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 717726 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-08T03:52:25.553Z
   [junit4]   2> 717728 INFO  (zkCallback-995-thread-1-processing-n:127.0.0.1:56662_) [n:127.0.0.1:56662_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 717730 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 717730 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/shard-2-001/solr.xml
   [junit4]   2> 717734 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5cb7ef52, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 717737 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34126/solr
   [junit4]   2> 717749 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:55556_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 717751 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:55556_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 717756 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:55556_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55556_
   [junit4]   2> 717757 INFO  (zkCallback-995-thread-1-processing-n:127.0.0.1:56662_) [n:127.0.0.1:56662_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 717757 INFO  (zkCallback-990-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 717758 INFO  (zkCallback-983-thread-2-processing-n:127.0.0.1:47631_) [n:127.0.0.1:47631_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 717765 INFO  (zkCallback-1001-thread-1-processing-n:127.0.0.1:55556_) [n:127.0.0.1:55556_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 717866 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:55556_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5cb7ef52
   [junit4]   2> 717870 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:55556_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5cb7ef52
   [junit4]   2> 717870 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:55556_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5cb7ef52
   [junit4]   2> 717871 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:55556_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/shard-2-001/cores
   [junit4]   2> 717894 INFO  (qtp572372145-8808) [n:127.0.0.1:55556_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:55556_&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 717897 INFO  (OverseerThreadFactory-2680-thread-4-processing-n:127.0.0.1:47631_) [n:127.0.0.1:47631_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:55556_ for creating new replica
   [junit4]   2> 717899 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t0&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 717899 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 718002 INFO  (zkCallback-995-thread-1-processing-n:127.0.0.1:56662_) [n:127.0.0.1:56662_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 718002 INFO  (zkCallback-1001-thread-1-processing-n:127.0.0.1:55556_) [n:127.0.0.1:55556_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 718911 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 718922 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.s.IndexSchema [collection1_shard1_replica_t0] Schema name=test
   [junit4]   2> 719002 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 719015 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t0' using configuration from collection collection1, trusted=true
   [junit4]   2> 719015 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_t0' (registry 'solr.core.collection1.shard1.replica_t0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5cb7ef52
   [junit4]   2> 719015 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 719015 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.c.SolrCore [[collection1_shard1_replica_t0] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/shard-2-001/cores/collection1_shard1_replica_t0], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/shard-2-001/cores/collection1_shard1_replica_t0/data/]
   [junit4]   2> 719018 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=10, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 719025 WARN  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 719057 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 719057 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 719059 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
   [junit4]   2> 719059 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 719060 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@3f2cf4e1
   [junit4]   2> 719060 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.s.SolrIndexSearcher Opening [Searcher@696821d9[collection1_shard1_replica_t0] main]
   [junit4]   2> 719062 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 719062 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 719063 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 719063 INFO  (searcherExecutor-2705-thread-1-processing-n:127.0.0.1:55556_ x:collection1_shard1_replica_t0 s:shard1 c:collection1) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.c.SolrCore [collection1_shard1_replica_t0] Registered new searcher Searcher@696821d9[collection1_shard1_replica_t0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 719064 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572324976247177216
   [junit4]   2> 719070 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 719070 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 719070 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:55556/collection1_shard1_replica_t0/
   [junit4]   2> 719070 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 719070 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.c.SyncStrategy http://127.0.0.1:55556/collection1_shard1_replica_t0/ has no replicas
   [junit4]   2> 719070 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 719071 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.c.ZkController collection1_shard1_replica_t0 stopping background replication from leader
   [junit4]   2> 719073 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:55556/collection1_shard1_replica_t0/ shard1
   [junit4]   2> 719175 INFO  (zkCallback-995-thread-1-processing-n:127.0.0.1:56662_) [n:127.0.0.1:56662_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 719175 INFO  (zkCallback-1001-thread-1-processing-n:127.0.0.1:55556_) [n:127.0.0.1:55556_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 719224 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 719225 INFO  (qtp572372145-8806) [n:127.0.0.1:55556_ c:collection1 s:shard1  x:collection1_shard1_replica_t0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t0&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1326
   [junit4]   2> 719228 INFO  (qtp572372145-8808) [n:127.0.0.1:55556_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:55556_&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2} status=0 QTime=1334
   [junit4]   2> 719328 INFO  (zkCallback-1001-thread-1-processing-n:127.0.0.1:55556_) [n:127.0.0.1:55556_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 719328 INFO  (zkCallback-995-thread-1-processing-n:127.0.0.1:56662_) [n:127.0.0.1:56662_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 719331 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/shard-3-001 of type PULL
   [junit4]   2> 719332 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 719333 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@61bdd272{/,null,AVAILABLE}
   [junit4]   2> 719334 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@13102516{HTTP/1.1,[http/1.1]}{127.0.0.1:46256}
   [junit4]   2> 719334 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.e.j.s.Server Started @721523ms
   [junit4]   2> 719334 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/tempDir-001/jetty3, replicaType=PULL, solrconfig=solrconfig.xml, hostContext=/, hostPort=46256, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/shard-3-001/cores}
   [junit4]   2> 719334 ERROR (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 719335 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 719335 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 719335 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 719335 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-08T03:52:27.162Z
   [junit4]   2> 719337 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 719337 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_CAA5C5599C244612-001/shard-3-001/solr.xml
   [junit4]   2> 719341 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5cb7ef52, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 719343 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34126/solr
   [junit4]   2> 719351 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:46256_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 719353 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA5C5599C244612]) [n:127.0.0.1:46256_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 719355 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[CAA

[...truncated too long message...]

.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   [junit4]    >    6) Thread[id=8907, name=StoppableCommitThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:502)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
   [junit4]    >         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)
   [junit4]    >         at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
   [junit4]    >         at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
   [junit4]    >         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]    >         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
   [junit4]    >         at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1124)
   [junit4]    >         at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1110)
   [junit4]    >         at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:646)
   [junit4]    >         at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1227)
   [junit4]    >         at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)
   [junit4]    >         at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
   [junit4]    >         at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)
   [junit4]    >         at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)
   [junit4]    >         at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:49)
   [junit4]    >    7) Thread[id=10455, name=zkCallback-1016-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    8) Thread[id=10457, name=zkCallback-1016-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    9) Thread[id=10456, name=zkCallback-1016-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([CAA5C5599C244612]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=10416, name=StoppableCommitThread-SendThread(127.0.0.1:34126), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   [junit4]    >    2) Thread[id=10418, name=zkCallback-1016-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    3) Thread[id=10455, name=zkCallback-1016-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    4) Thread[id=10457, name=zkCallback-1016-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    5) Thread[id=10456, name=zkCallback-1016-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([CAA5C5599C244612]:0)
   [junit4] Completed [179/727 (1!)] on J0 in 86.01s, 1 test, 1 failure, 2 errors <<< FAILURES!

[...truncated 45038 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 5 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/5/

2 tests failed.
FAILED:  org.apache.lucene.codecs.memory.TestMemoryDocValuesFormat.testSparseSortedFixedLengthVsStoredFields

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([623D4CC06F663EA4]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.codecs.memory.TestMemoryDocValuesFormat

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([623D4CC06F663EA4]:0)




Build Log:
[...truncated 6731 lines...]
   [junit4] Suite: org.apache.lucene.codecs.memory.TestMemoryDocValuesFormat
   [junit4]   2> 7월 09, 2017 4:19:29 오전 com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.codecs.memory.TestMemoryDocValuesFormat
   [junit4]   2>    1) Thread[id=11, name=SUITE-TestMemoryDocValuesFormat-seed#[623D4CC06F663EA4], state=RUNNABLE, group=TGRP-TestMemoryDocValuesFormat]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1559)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:696)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:693)
   [junit4]   2>         at java.security.AccessController.doPrivileged(Native Method)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:678)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:598)
   [junit4]   2>    2) Thread[id=9, name=JUnit4-serializer-daemon, state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
   [junit4]   2>    3) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1252)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1326)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:608)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:457)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:251)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:368)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2>    4) Thread[id=12, name=TEST-TestMemoryDocValuesFormat.testSparseSortedFixedLengthVsStoredFields-seed#[623D4CC06F663EA4], state=TIMED_WAITING, group=TGRP-TestMemoryDocValuesFormat]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4628)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:2091)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:2019)
   [junit4]   2>         at org.apache.lucene.index.RandomIndexWriter.forceMerge(RandomIndexWriter.java:429)
   [junit4]   2>         at org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedVsStoredFields(BaseDocValuesFormatTestCase.java:1591)
   [junit4]   2>         at org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedVsStoredFields(BaseDocValuesFormatTestCase.java:1648)
   [junit4]   2>         at org.apache.lucene.index.BaseDocValuesFormatTestCase.testSparseSortedFixedLengthVsStoredFields(BaseDocValuesFormatTestCase.java:1629)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestMemoryDocValuesFormat -Dtests.method=testSparseSortedFixedLengthVsStoredFields -Dtests.seed=623D4CC06F663EA4 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=ko-KR -Dtests.timezone=Egypt -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   7014s J0 | TestMemoryDocValuesFormat.testSparseSortedFixedLengthVsStoredFields <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > at __randomizedtesting.SeedInfo.seed([623D4CC06F663EA4]:0)
   [junit4]   2> 7월 09, 2017 4:19:29 오전 com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> 7월 09, 2017 4:19:49 오전 com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.lucene.codecs.memory.TestMemoryDocValuesFormat:
   [junit4]   2>    1) Thread[id=12, name=TEST-TestMemoryDocValuesFormat.testSparseSortedFixedLengthVsStoredFields-seed#[623D4CC06F663EA4], state=TIMED_WAITING, group=TGRP-TestMemoryDocValuesFormat]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4628)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:2091)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:2019)
   [junit4]   2>         at org.apache.lucene.index.RandomIndexWriter.forceMerge(RandomIndexWriter.java:429)
   [junit4]   2>         at org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedVsStoredFields(BaseDocValuesFormatTestCase.java:1591)
   [junit4]   2>         at org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedVsStoredFields(BaseDocValuesFormatTestCase.java:1648)
   [junit4]   2>         at org.apache.lucene.index.BaseDocValuesFormatTestCase.testSparseSortedFixedLengthVsStoredFields(BaseDocValuesFormatTestCase.java:1629)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 7월 09, 2017 4:19:49 오전 com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=12, name=TEST-TestMemoryDocValuesFormat.testSparseSortedFixedLengthVsStoredFields-seed#[623D4CC06F663EA4], state=TIMED_WAITING, group=TGRP-TestMemoryDocValuesFormat]
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/codecs/test/J0/temp/lucene.codecs.memory.TestMemoryDocValuesFormat_623D4CC06F663EA4-001
   [junit4]   2> Jul 09, 2017 2:19:49 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=800, maxMBSortInHeap=6.291068436637745, sim=RandomSimilarity(queryNorm=false): {fieldname=DFR I(n)B2}, locale=ko-KR, timezone=Egypt
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=213825664,total=319291392
   [junit4]   2> NOTE: All tests run in this JVM: [TestMemoryDocValuesFormat]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestMemoryDocValuesFormat -Dtests.seed=623D4CC06F663EA4 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=ko-KR -Dtests.timezone=Egypt -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | TestMemoryDocValuesFormat (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > at __randomizedtesting.SeedInfo.seed([623D4CC06F663EA4]:0)
   [junit4] Completed [18/20 (1!)] on J0 in 7220.46s, 59 tests, 2 errors <<< FAILURES!

[...truncated 50330 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 6 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/6/

No tests ran.

Build Log:
[...truncated 79 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:818: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/common-build.xml:422: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/common-build.xml:459: Ivy is not available

Total time: 0 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 7 - Still Failing

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/7/

6 tests failed.
FAILED:  org.apache.lucene.index.TestIndexSorting.testRandom3

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([33668C7C0492929E]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexSorting

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([33668C7C0492929E]:0)


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649)
        at java.lang.StringBuilder.append(StringBuilder.java:202)
        at org.apache.http.client.utils.URLEncodedUtils.urlEncode(URLEncodedUtils.java:522)
        at org.apache.http.client.utils.URLEncodedUtils.encodeFormFields(URLEncodedUtils.java:638)
        at org.apache.http.client.utils.URLEncodedUtils.format(URLEncodedUtils.java:390)
        at org.apache.http.client.utils.URLEncodedUtils.format(URLEncodedUtils.java:369)
        at org.apache.http.client.entity.UrlEncodedFormEntity.<init>(UrlEncodedFormEntity.java:77)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:447)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:974)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:990)
        at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:228)
        at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:167)
        at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:668)
        at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
20 threads leaked from SUITE scope at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest:     1) Thread[id=20390, name=searcherExecutor-5760-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=20080, name=qtp1812211277-20080, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at org.apache.log4j.Category.callAppenders(Category.java:204)         at org.apache.log4j.Category.forcedLog(Category.java:391)         at org.apache.log4j.Category.log(Category.java:856)         at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)         at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=20402, name=searcherExecutor-5763-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=20389, name=searcherExecutor-5761-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=20111, name=qtp150412892-20111, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140)         at org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:110)         at org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:138)         at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState.document(CompressingStoredFieldsReader.java:555)         at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.document(CompressingStoredFieldsReader.java:571)         at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:578)         at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat$AssertingStoredFieldsReader.visitDocument(AssertingStoredFieldsFormat.java:73)         at org.apache.lucene.index.CodecReader.document(CodecReader.java:83)         at org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:341)         at org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:341)         at org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:118)         at org.apache.solr.search.SolrDocumentFetcher.doc(SolrDocumentFetcher.java:198)         at org.apache.solr.response.DocsStreamer.next(DocsStreamer.java:155)         at org.apache.solr.response.DocsStreamer.next(DocsStreamer.java:59)         at org.apache.solr.response.BinaryResponseWriter$Resolver.writeResultsBody(BinaryResponseWriter.java:126)         at org.apache.solr.response.BinaryResponseWriter$Resolver.writeResults(BinaryResponseWriter.java:145)         at org.apache.solr.response.BinaryResponseWriter$Resolver.resolve(BinaryResponseWriter.java:89)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:239)         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)         at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:52)         at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:49)         at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:806)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=20398, name=searcherExecutor-5765-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=20525, name=qtp150412892-20525, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at java.io.FilterOutputStream.write(FilterOutputStream.java:125)         at org.apache.lucene.util.TestRuleLimitSysouts$DelegateStream.write(TestRuleLimitSysouts.java:128)         at java.io.PrintStream.write(PrintStream.java:480)         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)         at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)         at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)         at java.io.Writer.write(Writer.java:157)         at org.apache.log4j.helpers.QuietWriter.write(QuietWriter.java:48)         at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)         at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)         at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)         at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)         at org.apache.log4j.Category.callAppenders(Category.java:206)         at org.apache.log4j.Category.forcedLog(Category.java:391)         at org.apache.log4j.Category.log(Category.java:856)         at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)         at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)    8) Thread[id=20514, name=qtp423659769-20514, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at org.apache.log4j.Category.callAppenders(Category.java:204)         at org.apache.log4j.Category.forcedLog(Category.java:391)         at org.apache.log4j.Category.log(Category.java:856)         at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)         at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)    9) Thread[id=20489, name=qtp423659769-20489, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at org.apache.log4j.Category.callAppenders(Category.java:204)         at org.apache.log4j.Category.forcedLog(Category.java:391)         at org.apache.log4j.Category.log(Category.java:856)         at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:381)         at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:810)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   10) Thread[id=20501, name=qtp1812211277-20501, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at java.util.Arrays.copyOfRange(Arrays.java:3664)         at java.lang.String.<init>(String.java:207)         at java.lang.StringBuilder.toString(StringBuilder.java:407)         at org.apache.lucene.store.MockIndexInputWrapper.<init>(MockIndexInputWrapper.java:40)         at org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:89)         at org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:29)         at org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader$SegmentTermsEnum.<init>(BlockTermsReader.java:331)         at org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader.iterator(BlockTermsReader.java:243)         at org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:121)         at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         at org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)         at org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)         at org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:655)         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)         at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)         at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1591)         at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1408)         at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:575)         at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:544)         at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2473)         at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   11) Thread[id=20400, name=searcherExecutor-5773-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)   12) Thread[id=20401, name=searcherExecutor-5767-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)   13) Thread[id=20393, name=searcherExecutor-5769-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)   14) Thread[id=20410, name=searcherExecutor-5771-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)   15) Thread[id=20166, name=qtp2106581206-20166, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at java.lang.String.valueOf(String.java:2994)         at java.lang.StringBuilder.append(StringBuilder.java:131)         at org.apache.lucene.store.MockIndexInputWrapper.<init>(MockIndexInputWrapper.java:40)         at org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:89)         at org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:29)         at org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader$SegmentTermsEnum.<init>(BlockTermsReader.java:331)         at org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader.iterator(BlockTermsReader.java:243)         at org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:121)         at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         at org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)         at org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)         at org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:655)         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)         at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)         at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1591)         at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1408)         at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:575)         at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:544)         at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2473)         at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   16) Thread[id=20408, name=searcherExecutor-5776-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)   17) Thread[id=20543, name=qtp1812211277-20543, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at org.apache.lucene.codecs.blockterms.VariableGapTermsIndexReader$IndexEnum.<init>(VariableGapTermsIndexReader.java:97)         at org.apache.lucene.codecs.blockterms.VariableGapTermsIndexReader.getFieldEnum(VariableGapTermsIndexReader.java:189)         at org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader$SegmentTermsEnum.<init>(BlockTermsReader.java:333)         at org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader.iterator(BlockTermsReader.java:243)         at org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:121)         at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         at org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)         at org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)         at org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:655)         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)         at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)         at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1591)         at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1408)         at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:575)         at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:544)         at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2473)         at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   18) Thread[id=20397, name=searcherExecutor-5781-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)   19) Thread[id=20495, name=qtp423659769-20495, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at org.apache.log4j.Category.callAppenders(Category.java:204)         at org.apache.log4j.Category.forcedLog(Category.java:391)         at org.apache.log4j.Category.log(Category.java:856)         at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)         at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   20) Thread[id=20512, name=qtp150412892-20512, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at org.apache.log4j.Category.callAppenders(Category.java:204)         at org.apache.log4j.Category.forcedLog(Category.java:391)         at org.apache.log4j.Category.log(Category.java:856)         at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)         at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 20 threads leaked from SUITE scope at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest:
   1) Thread[id=20390, name=searcherExecutor-5760-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=20080, name=qtp1812211277-20080, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at org.apache.log4j.Category.callAppenders(Category.java:204)
        at org.apache.log4j.Category.forcedLog(Category.java:391)
        at org.apache.log4j.Category.log(Category.java:856)
        at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=20402, name=searcherExecutor-5763-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=20389, name=searcherExecutor-5761-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=20111, name=qtp150412892-20111, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140)
        at org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:110)
        at org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:138)
        at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState.document(CompressingStoredFieldsReader.java:555)
        at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.document(CompressingStoredFieldsReader.java:571)
        at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:578)
        at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat$AssertingStoredFieldsReader.visitDocument(AssertingStoredFieldsFormat.java:73)
        at org.apache.lucene.index.CodecReader.document(CodecReader.java:83)
        at org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:341)
        at org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:341)
        at org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:118)
        at org.apache.solr.search.SolrDocumentFetcher.doc(SolrDocumentFetcher.java:198)
        at org.apache.solr.response.DocsStreamer.next(DocsStreamer.java:155)
        at org.apache.solr.response.DocsStreamer.next(DocsStreamer.java:59)
        at org.apache.solr.response.BinaryResponseWriter$Resolver.writeResultsBody(BinaryResponseWriter.java:126)
        at org.apache.solr.response.BinaryResponseWriter$Resolver.writeResults(BinaryResponseWriter.java:145)
        at org.apache.solr.response.BinaryResponseWriter$Resolver.resolve(BinaryResponseWriter.java:89)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:239)
        at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:52)
        at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:49)
        at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:806)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=20398, name=searcherExecutor-5765-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=20525, name=qtp150412892-20525, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at java.io.FilterOutputStream.write(FilterOutputStream.java:125)
        at org.apache.lucene.util.TestRuleLimitSysouts$DelegateStream.write(TestRuleLimitSysouts.java:128)
        at java.io.PrintStream.write(PrintStream.java:480)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
        at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
        at java.io.Writer.write(Writer.java:157)
        at org.apache.log4j.helpers.QuietWriter.write(QuietWriter.java:48)
        at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)
        at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
        at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
        at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
        at org.apache.log4j.Category.callAppenders(Category.java:206)
        at org.apache.log4j.Category.forcedLog(Category.java:391)
        at org.apache.log4j.Category.log(Category.java:856)
        at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=20514, name=qtp423659769-20514, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at org.apache.log4j.Category.callAppenders(Category.java:204)
        at org.apache.log4j.Category.forcedLog(Category.java:391)
        at org.apache.log4j.Category.log(Category.java:856)
        at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
   9) Thread[id=20489, name=qtp423659769-20489, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at org.apache.log4j.Category.callAppenders(Category.java:204)
        at org.apache.log4j.Category.forcedLog(Category.java:391)
        at org.apache.log4j.Category.log(Category.java:856)
        at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:381)
        at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:810)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
  10) Thread[id=20501, name=qtp1812211277-20501, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at java.util.Arrays.copyOfRange(Arrays.java:3664)
        at java.lang.String.<init>(String.java:207)
        at java.lang.StringBuilder.toString(StringBuilder.java:407)
        at org.apache.lucene.store.MockIndexInputWrapper.<init>(MockIndexInputWrapper.java:40)
        at org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:89)
        at org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:29)
        at org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader$SegmentTermsEnum.<init>(BlockTermsReader.java:331)
        at org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader.iterator(BlockTermsReader.java:243)
        at org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:121)
        at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)
        at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)
        at org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
        at org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)
        at org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:655)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
        at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
        at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1591)
        at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1408)
        at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:575)
        at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:544)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2473)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
  11) Thread[id=20400, name=searcherExecutor-5773-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
  12) Thread[id=20401, name=searcherExecutor-5767-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
  13) Thread[id=20393, name=searcherExecutor-5769-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
  14) Thread[id=20410, name=searcherExecutor-5771-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
  15) Thread[id=20166, name=qtp2106581206-20166, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at java.lang.String.valueOf(String.java:2994)
        at java.lang.StringBuilder.append(StringBuilder.java:131)
        at org.apache.lucene.store.MockIndexInputWrapper.<init>(MockIndexInputWrapper.java:40)
        at org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:89)
        at org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:29)
        at org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader$SegmentTermsEnum.<init>(BlockTermsReader.java:331)
        at org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader.iterator(BlockTermsReader.java:243)
        at org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:121)
        at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)
        at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)
        at org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
        at org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)
        at org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:655)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
        at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
        at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1591)
        at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1408)
        at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:575)
        at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:544)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2473)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
  16) Thread[id=20408, name=searcherExecutor-5776-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
  17) Thread[id=20543, name=qtp1812211277-20543, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at org.apache.lucene.codecs.blockterms.VariableGapTermsIndexReader$IndexEnum.<init>(VariableGapTermsIndexReader.java:97)
        at org.apache.lucene.codecs.blockterms.VariableGapTermsIndexReader.getFieldEnum(VariableGapTermsIndexReader.java:189)
        at org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader$SegmentTermsEnum.<init>(BlockTermsReader.java:333)
        at org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader.iterator(BlockTermsReader.java:243)
        at org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:121)
        at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)
        at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)
        at org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
        at org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)
        at org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:655)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
        at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
        at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1591)
        at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1408)
        at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:575)
        at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:544)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2473)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
  18) Thread[id=20397, name=searcherExecutor-5781-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
  19) Thread[id=20495, name=qtp423659769-20495, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at org.apache.log4j.Category.callAppenders(Category.java:204)
        at org.apache.log4j.Category.forcedLog(Category.java:391)
        at org.apache.log4j.Category.log(Category.java:856)
        at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java

[...truncated too long message...]

.handle(GzipHandler.java:395)
   [junit4]    >         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]    >         at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]    >         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]    >         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]    >         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]    >         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]    >         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   14) Thread[id=20495, name=qtp423659769-20495, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]    >         at org.apache.log4j.Category.callAppenders(Category.java:204)
   [junit4]    >         at org.apache.log4j.Category.forcedLog(Category.java:391)
   [junit4]    >         at org.apache.log4j.Category.log(Category.java:856)
   [junit4]    >         at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:381)
   [junit4]    >         at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:810)
   [junit4]    >         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)
   [junit4]    >         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
   [junit4]    >         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
   [junit4]    >         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]    >         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]    >         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]    >         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]    >         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]    >         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]    >         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]    >         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]    >         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]    >         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]    >         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
   [junit4]    >         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]    >         at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]    >         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]    >         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]    >         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]    >         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]    >         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   15) Thread[id=20512, name=qtp150412892-20512, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]    >         at org.apache.log4j.Category.callAppenders(Category.java:204)
   [junit4]    >         at org.apache.log4j.Category.forcedLog(Category.java:391)
   [junit4]    >         at org.apache.log4j.Category.log(Category.java:856)
   [junit4]    >         at org.slf4j.impl.Log4jLoggerAdapter.warn(Log4jLoggerAdapter.java:400)
   [junit4]    >         at org.apache.solr.metrics.SolrMetricManager.closeReporters(SolrMetricManager.java:943)
   [junit4]    >         at org.apache.solr.metrics.SolrCoreMetricManager.close(SolrCoreMetricManager.java:146)
   [junit4]    >         at org.apache.solr.core.SolrCore.close(SolrCore.java:1581)
   [junit4]    >         at org.apache.solr.servlet.HttpSolrCall.destroy(HttpSolrCall.java:579)
   [junit4]    >         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:391)
   [junit4]    >         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
   [junit4]    >         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]    >         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]    >         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]    >         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]    >         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]    >         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]    >         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]    >         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]    >         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]    >         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]    >         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
   [junit4]    >         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]    >         at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]    >         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]    >         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]    >         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]    >         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]    >         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([312D14671E6B3DDE]:0)
   [junit4] Completed [505/727 (2!)] on J0 in 1699.93s, 1 test, 3 errors <<< FAILURES!

[...truncated 675 lines...]
   [junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J0-20170712_045056_5712924593271098990980.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/heapdumps/java_pid18990.hprof ...
   [junit4] Heap dump file created [675978761 bytes in 17.155 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 7505 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:818: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:762: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid18990.hprof

Total time: 376 minutes 52 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 8 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/8/

5 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([57BDCF540E9917CD:DFE9F08EA0657A35]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.update.AutoCommitTest.testCommitWithin

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
        at __randomizedtesting.SeedInfo.seed([57BDCF540E9917CD:ED6FA02C8DB7F9D8]:0)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:878)
        at org.apache.solr.update.AutoCommitTest.testCommitWithin(AutoCommitTest.java:353)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
        xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

        request was:q=id:529&qt=&start=0&rows=20&version=2.2
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:871)
        ... 40 more


FAILED:  org.apache.solr.update.AutoCommitTest.testMaxTime

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
        at __randomizedtesting.SeedInfo.seed([57BDCF540E9917CD:CD49B2B690038BF1]:0)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:878)
        at org.apache.solr.update.AutoCommitTest.testMaxTime(AutoCommitTest.java:270)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
        xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

        request was:q=id:529&qt=&start=0&rows=20&version=2.2
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:871)
        ... 40 more


FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
Didn't see all replicas for shard shard1 in c8n_1x2 come up within 90000 ms! ClusterState: {   "collMinRf_1x3":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{"shard1":{         "range":"80000000-7fffffff",         "state":"active",         "replicas":{           "core_node1":{             "core":"collMinRf_1x3_shard1_replica_n1",             "base_url":"http://127.0.0.1:36537",             "node_name":"127.0.0.1:36537_",             "state":"active",             "type":"NRT"},           "core_node2":{             "core":"collMinRf_1x3_shard1_replica_n2",             "base_url":"http://127.0.0.1:35393",             "node_name":"127.0.0.1:35393_",             "state":"active",             "type":"NRT"},           "core_node3":{             "core":"collMinRf_1x3_shard1_replica_n3",             "base_url":"http://127.0.0.1:36929",             "node_name":"127.0.0.1:36929_",             "state":"active",             "type":"NRT",             "leader":"true"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"3",     "tlogReplicas":"0"},   "collection1":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{       "shard1":{         "range":"80000000-ffffffff",         "state":"active",         "replicas":{"core_node2":{             "core":"collection1_shard1_replica_n0",             "base_url":"http://127.0.0.1:41363",             "node_name":"127.0.0.1:41363_",             "state":"active",             "type":"NRT",             "leader":"true"}}},       "shard2":{         "range":"0-7fffffff",         "state":"active",         "replicas":{           "core_node1":{             "core":"collection1_shard2_replica_n0",             "base_url":"http://127.0.0.1:36929",             "node_name":"127.0.0.1:36929_",             "state":"active",             "type":"NRT",             "leader":"true"},           "core_node3":{             "core":"collection1_shard2_replica_n1",             "base_url":"http://127.0.0.1:35393",             "node_name":"127.0.0.1:35393_",             "state":"active",             "type":"NRT"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"1",     "tlogReplicas":"0"},   "control_collection":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{"shard1":{         "range":"80000000-7fffffff",         "state":"active",         "replicas":{"core_node1":{             "core":"control_collection_shard1_replica_n1",             "base_url":"http://127.0.0.1:36537",             "node_name":"127.0.0.1:36537_",             "state":"active",             "type":"NRT",             "leader":"true"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"1",     "tlogReplicas":"0"},   "c8n_1x2":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{"shard1":{         "range":"80000000-7fffffff",         "state":"active",         "replicas":{           "core_node1":{             "core":"c8n_1x2_shard1_replica_n1",             "base_url":"http://127.0.0.1:36929",             "node_name":"127.0.0.1:36929_",             "state":"active",             "type":"NRT",             "leader":"true"},           "core_node2":{             "core":"c8n_1x2_shard1_replica_n2",             "base_url":"http://127.0.0.1:41363",             "node_name":"127.0.0.1:41363_",             "state":"recovering",             "type":"NRT"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"2",     "tlogReplicas":"0"}}

Stack Trace:
java.lang.AssertionError: Didn't see all replicas for shard shard1 in c8n_1x2 come up within 90000 ms! ClusterState: {
  "collMinRf_1x3":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{
          "core_node1":{
            "core":"collMinRf_1x3_shard1_replica_n1",
            "base_url":"http://127.0.0.1:36537",
            "node_name":"127.0.0.1:36537_",
            "state":"active",
            "type":"NRT"},
          "core_node2":{
            "core":"collMinRf_1x3_shard1_replica_n2",
            "base_url":"http://127.0.0.1:35393",
            "node_name":"127.0.0.1:35393_",
            "state":"active",
            "type":"NRT"},
          "core_node3":{
            "core":"collMinRf_1x3_shard1_replica_n3",
            "base_url":"http://127.0.0.1:36929",
            "node_name":"127.0.0.1:36929_",
            "state":"active",
            "type":"NRT",
            "leader":"true"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"3",
    "tlogReplicas":"0"},
  "collection1":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{
      "shard1":{
        "range":"80000000-ffffffff",
        "state":"active",
        "replicas":{"core_node2":{
            "core":"collection1_shard1_replica_n0",
            "base_url":"http://127.0.0.1:41363",
            "node_name":"127.0.0.1:41363_",
            "state":"active",
            "type":"NRT",
            "leader":"true"}}},
      "shard2":{
        "range":"0-7fffffff",
        "state":"active",
        "replicas":{
          "core_node1":{
            "core":"collection1_shard2_replica_n0",
            "base_url":"http://127.0.0.1:36929",
            "node_name":"127.0.0.1:36929_",
            "state":"active",
            "type":"NRT",
            "leader":"true"},
          "core_node3":{
            "core":"collection1_shard2_replica_n1",
            "base_url":"http://127.0.0.1:35393",
            "node_name":"127.0.0.1:35393_",
            "state":"active",
            "type":"NRT"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"1",
    "tlogReplicas":"0"},
  "control_collection":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{"core_node1":{
            "core":"control_collection_shard1_replica_n1",
            "base_url":"http://127.0.0.1:36537",
            "node_name":"127.0.0.1:36537_",
            "state":"active",
            "type":"NRT",
            "leader":"true"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"1",
    "tlogReplicas":"0"},
  "c8n_1x2":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{
          "core_node1":{
            "core":"c8n_1x2_shard1_replica_n1",
            "base_url":"http://127.0.0.1:36929",
            "node_name":"127.0.0.1:36929_",
            "state":"active",
            "type":"NRT",
            "leader":"true"},
          "core_node2":{
            "core":"c8n_1x2_shard1_replica_n2",
            "base_url":"http://127.0.0.1:41363",
            "node_name":"127.0.0.1:41363_",
            "state":"recovering",
            "type":"NRT"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"2",
    "tlogReplicas":"0"}}
        at __randomizedtesting.SeedInfo.seed([57BDCF540E9917CD:DFE9F08EA0657A35]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.ensureAllReplicasAreActive(AbstractFullDistribZkTestBase.java:1992)
        at org.apache.solr.cloud.HttpPartitionTest.testRf2(HttpPartitionTest.java:367)
        at org.apache.solr.cloud.HttpPartitionTest.test(HttpPartitionTest.java:131)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([57BDCF540E9917CD:DFE9F08EA0657A35]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12100 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/init-core-data-001
   [junit4]   2> 880765 WARN  (SUITE-HttpPartitionTest-seed#[57BDCF540E9917CD]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 880765 INFO  (SUITE-HttpPartitionTest-seed#[57BDCF540E9917CD]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 880766 INFO  (SUITE-HttpPartitionTest-seed#[57BDCF540E9917CD]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 880766 INFO  (SUITE-HttpPartitionTest-seed#[57BDCF540E9917CD]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 880767 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 880768 INFO  (Thread-2272) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 880768 INFO  (Thread-2272) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 880769 ERROR (Thread-2272) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 880868 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.ZkTestServer start zk server on port:42343
   [junit4]   2> 880877 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 880878 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 880880 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 880880 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 880881 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 880882 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 880883 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 880884 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 880885 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 880886 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 880887 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 880888 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 880962 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 880963 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6c156a75{/,null,AVAILABLE}
   [junit4]   2> 880964 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@df0e738{HTTP/1.1,[http/1.1]}{127.0.0.1:40816}
   [junit4]   2> 880964 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.e.j.s.Server Started @883040ms
   [junit4]   2> 880964 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=36537, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/control-001/cores}
   [junit4]   2> 880964 ERROR (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 880964 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 880964 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 880964 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 880964 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-13T03:09:47.070Z
   [junit4]   2> 880966 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 880966 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/control-001/solr.xml
   [junit4]   2> 880971 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@28b35f7f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 880973 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42343/solr
   [junit4]   2> 881030 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:36537_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 881031 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:36537_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36537_
   [junit4]   2> 881031 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:36537_    ] o.a.s.c.Overseer Overseer (id=98298454794174468-127.0.0.1:36537_-n_0000000000) starting
   [junit4]   2> 881036 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:36537_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36537_
   [junit4]   2> 881037 INFO  (zkCallback-1903-thread-1-processing-n:127.0.0.1:36537_) [n:127.0.0.1:36537_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 881092 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:36537_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@28b35f7f
   [junit4]   2> 881096 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:36537_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@28b35f7f
   [junit4]   2> 881096 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:36537_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@28b35f7f
   [junit4]   2> 881097 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:36537_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/control-001/cores
   [junit4]   2> 881111 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 881112 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42343/solr ready
   [junit4]   2> 881113 INFO  (SocketProxy-Acceptor-36537) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=52942,localport=36537], receiveBufferSize:531000
   [junit4]   2> 881113 INFO  (SocketProxy-Acceptor-36537) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=40816,localport=34446], receiveBufferSize=530904
   [junit4]   2> 881115 INFO  (qtp1976280674-9003) [n:127.0.0.1:36537_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:36537_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 881117 INFO  (OverseerThreadFactory-3120-thread-1-processing-n:127.0.0.1:36537_) [n:127.0.0.1:36537_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 881220 INFO  (SocketProxy-Acceptor-36537) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=52946,localport=36537], receiveBufferSize:531000
   [junit4]   2> 881220 INFO  (SocketProxy-Acceptor-36537) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=40816,localport=34450], receiveBufferSize=530904
   [junit4]   2> 881223 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 881223 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 881326 INFO  (zkCallback-1903-thread-1-processing-n:127.0.0.1:36537_) [n:127.0.0.1:36537_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 882233 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 882244 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 882324 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 882335 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 882335 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@28b35f7f
   [junit4]   2> 882335 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 882335 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 882337 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=7, maxMergeAtOnceExplicit=5, maxMergedSegmentMB=8.29296875, floorSegmentMB=1.4375, forceMergeDeletesPctAllowed=7.88214974082647, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.477010148973108
   [junit4]   2> 882339 WARN  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 882370 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 882370 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 882371 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 882372 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 882372 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1005444240, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 882373 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2ace77c7[control_collection_shard1_replica_n1] main]
   [junit4]   2> 882374 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 882374 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 882374 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 882375 INFO  (searcherExecutor-3123-thread-1-processing-n:127.0.0.1:36537_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@2ace77c7[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 882375 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572775278391853056
   [junit4]   2> 882380 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 882380 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 882380 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:36537/control_collection_shard1_replica_n1/
   [junit4]   2> 882380 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 882380 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:36537/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 882380 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 882383 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:36537/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 882484 INFO  (zkCallback-1903-thread-1-processing-n:127.0.0.1:36537_) [n:127.0.0.1:36537_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 882533 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 882535 INFO  (qtp1976280674-9001) [n:127.0.0.1:36537_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1312
   [junit4]   2> 882544 INFO  (qtp1976280674-9003) [n:127.0.0.1:36537_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 882643 INFO  (zkCallback-1903-thread-1-processing-n:127.0.0.1:36537_) [n:127.0.0.1:36537_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 883119 INFO  (OverseerCollectionConfigSetProcessor-98298454794174468-127.0.0.1:36537_-n_0000000000) [n:127.0.0.1:36537_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 883545 INFO  (qtp1976280674-9003) [n:127.0.0.1:36537_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:36537_&wt=javabin&version=2} status=0 QTime=2430
   [junit4]   2> 883550 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 883550 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42343/solr ready
   [junit4]   2> 883550 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 883551 INFO  (SocketProxy-Acceptor-36537) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=52964,localport=36537], receiveBufferSize:531000
   [junit4]   2> 883551 INFO  (SocketProxy-Acceptor-36537) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=40816,localport=34468], receiveBufferSize=530904
   [junit4]   2> 883552 INFO  (qtp1976280674-9007) [n:127.0.0.1:36537_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 883554 INFO  (OverseerThreadFactory-3120-thread-2-processing-n:127.0.0.1:36537_) [n:127.0.0.1:36537_    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 883555 WARN  (OverseerThreadFactory-3120-thread-2-processing-n:127.0.0.1:36537_) [n:127.0.0.1:36537_    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 883758 INFO  (qtp1976280674-9007) [n:127.0.0.1:36537_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 883758 INFO  (qtp1976280674-9007) [n:127.0.0.1:36537_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=206
   [junit4]   2> 883832 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/shard-1-001 of type NRT
   [junit4]   2> 883833 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 883834 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4fbfce95{/,null,AVAILABLE}
   [junit4]   2> 883834 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3b542add{HTTP/1.1,[http/1.1]}{127.0.0.1:37851}
   [junit4]   2> 883834 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.e.j.s.Server Started @885911ms
   [junit4]   2> 883834 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=36929, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/shard-1-001/cores}
   [junit4]   2> 883834 ERROR (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 883834 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 883834 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 883834 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 883834 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-13T03:09:49.940Z
   [junit4]   2> 883836 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 883836 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/shard-1-001/solr.xml
   [junit4]   2> 883840 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@28b35f7f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 883842 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42343/solr
   [junit4]   2> 883848 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:36929_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 883850 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:36929_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 883851 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:36929_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36929_
   [junit4]   2> 883852 INFO  (zkCallback-1910-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 883852 INFO  (zkCallback-1903-thread-1-processing-n:127.0.0.1:36537_) [n:127.0.0.1:36537_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 883852 INFO  (zkCallback-1915-thread-1-processing-n:127.0.0.1:36929_) [n:127.0.0.1:36929_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 883904 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:36929_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@28b35f7f
   [junit4]   2> 883908 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:36929_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@28b35f7f
   [junit4]   2> 883908 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:36929_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@28b35f7f
   [junit4]   2> 883909 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:36929_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/shard-1-001/cores
   [junit4]   2> 883926 INFO  (SocketProxy-Acceptor-36929) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=46180,localport=36929], receiveBufferSize:531000
   [junit4]   2> 883926 INFO  (SocketProxy-Acceptor-36929) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=37851,localport=39518], receiveBufferSize=530904
   [junit4]   2> 883927 INFO  (qtp421365138-9051) [n:127.0.0.1:36929_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:36929_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 883929 INFO  (OverseerCollectionConfigSetProcessor-98298454794174468-127.0.0.1:36537_-n_0000000000) [n:127.0.0.1:36537_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 883929 INFO  (OverseerThreadFactory-3120-thread-3-processing-n:127.0.0.1:36537_) [n:127.0.0.1:36537_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:36929_ for creating new replica
   [junit4]   2> 883930 INFO  (SocketProxy-Acceptor-36929) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=46184,localport=36929], receiveBufferSize:531000
   [junit4]   2> 883930 INFO  (SocketProxy-Acceptor-36929) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=37851,localport=39522], receiveBufferSize=530904
   [junit4]   2> 883930 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 883931 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 884033 INFO  (zkCallback-1915-thread-1-processing-n:127.0.0.1:36929_) [n:127.0.0.1:36929_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 884941 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 884951 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema [collection1_shard2_replica_n0] Schema name=test
   [junit4]   2> 885034 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 885046 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n0' using configuration from collection collection1, trusted=true
   [junit4]   2> 885046 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n0' (registry 'solr.core.collection1.shard2.replica_n0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@28b35f7f
   [junit4]   2> 885046 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 885046 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore [[collection1_shard2_replica_n0] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/shard-1-001/cores/collection1_shard2_replica_n0], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/shard-1-001/cores/collection1_shard2_replica_n0/data/]
   [junit4]   2> 885048 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=7, maxMergeAtOnceExplicit=5, maxMergedSegmentMB=8.29296875, floorSegmentMB=1.4375, forceMergeDeletesPctAllowed=7.88214974082647, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.477010148973108
   [junit4]   2> 885050 WARN  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 885082 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 885082 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 885083 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 885083 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 885084 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1005444240, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 885084 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.SolrIndexSearcher Opening [Searcher@721f0441[collection1_shard2_replica_n0] main]
   [junit4]   2> 885085 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 885086 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 885086 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 885087 INFO  (searcherExecutor-3134-thread-1-processing-n:127.0.0.1:36929_ x:collection1_shard2_replica_n0 s:shard2 c:collection1) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore [collection1_shard2_replica_n0] Registered new searcher Searcher@721f0441[collection1_shard2_replica_n0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 885087 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572775281235591168
   [junit4]   2> 885095 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 885095 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 885095 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:36929/collection1_shard2_replica_n0/
   [junit4]   2> 885095 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 885095 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy http://127.0.0.1:36929/collection1_shard2_replica_n0/ has no replicas
   [junit4]   2> 885095 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 885097 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:36929/collection1_shard2_replica_n0/ shard2
   [junit4]   2> 885199 INFO  (zkCallback-1915-thread-1-processing-n:127.0.0.1:36929_) [n:127.0.0.1:36929_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 885248 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 885250 INFO  (qtp421365138-9053) [n:127.0.0.1:36929_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1319
   [junit4]   2> 885251 INFO  (qtp421365138-9051) [n:127.0.0.1:36929_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:36929_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1324
   [junit4]   2> 885327 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/shard-2-001 of type NRT
   [junit4]   2> 885327 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 885328 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5bb73dfd{/,null,AVAILABLE}
   [junit4]   2> 885328 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7a01e966{HTTP/1.1,[http/1.1]}{127.0.0.1:32870}
   [junit4]   2> 885328 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.e.j.s.Server Started @887405ms
   [junit4]   2> 885328 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=41363, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/shard-2-001/cores}
   [junit4]   2> 885328 ERROR (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 885329 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 885329 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 885329 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 885329 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-13T03:09:51.435Z
   [junit4]   2> 885331 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 885331 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/shard-2-001/solr.xml
   [junit4]   2> 885335 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@28b35f7f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 885337 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42343/solr
   [junit4]   2> 885343 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:41363_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 885344 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:41363_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 885345 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:41363_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41363_
   [junit4]   2> 885346 INFO  (zkCallback-1903-thread-1-processing-n:127.0.0.1:36537_) [n:127.0.0.1:36537_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 885346 INFO  (zkCallback-1915-thread-1-processing-n:127.0.0.1:36929_) [n:127.0.0.1:36929_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 885346 INFO  (zkCallback-1910-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 885347 INFO  (zkCallback-1921-thread-1-processing-n:127.0.0.1:41363_) [n:127.0.0.1:41363_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 885448 INFO  (zkCallback-1915-thread-1-processing-n:127.0.0.1:36929_) [n:127.0.0.1:36929_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 885502 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:41363_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@28b35f7f
   [junit4]   2> 885506 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:41363_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@28b35f7f
   [junit4]   2> 885506 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:41363_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@28b35f7f
   [junit4]   2> 885507 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [n:127.0.0.1:41363_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/shard-2-001/cores
   [junit4]   2> 885536 INFO  (SocketProxy-Acceptor-41363) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=37364,localport=41363], receiveBufferSize:531000
   [junit4]   2> 885537 INFO  (SocketProxy-Acceptor-41363) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=32870,localport=47114], receiveBufferSize=530904
   [junit4]   2> 885538 INFO  (qtp87552865-9085) [n:127.0.0.1:41363_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:41363_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 885540 INFO  (OverseerCollectionConfigSetProcessor-98298454794174468-127.0.0.1:36537_-n_0000000000) [n:127.0.0.1:36537_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 885540 INFO  (OverseerThreadFactory-3120-thread-4-processing-n:127.0.0.1:36537_) [n:127.0.0.1:36537_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:41363_ for creating new replica
   [junit4]   2> 885541 INFO  (SocketProxy-Acceptor-41363) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=37368,localport=41363], receiveBufferSize:531000
   [junit4]   2> 885541 INFO  (SocketProxy-Acceptor-41363) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=32870,localport=47118], receiveBufferSize=530904
   [junit4]   2> 885542 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n0&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 885542 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 885645 INFO  (zkCallback-1915-thread-1-processing-n:127.0.0.1:36929_) [n:127.0.0.1:36929_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 885645 INFO  (zkCallback-1921-thread-1-processing-n:127.0.0.1:41363_) [n:127.0.0.1:41363_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 886553 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 886563 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.IndexSchema [collection1_shard1_replica_n0] Schema name=test
   [junit4]   2> 886639 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 886650 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n0' using configuration from collection collection1, trusted=true
   [junit4]   2> 886650 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n0' (registry 'solr.core.collection1.shard1.replica_n0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@28b35f7f
   [junit4]   2> 886650 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 886651 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrCore [[collection1_shard1_replica_n0] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/shard-2-001/cores/collection1_shard1_replica_n0], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/shard-2-001/cores/collection1_shard1_replica_n0/data/]
   [junit4]   2> 886652 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=7, maxMergeAtOnceExplicit=5, maxMergedSegmentMB=8.29296875, floorSegmentMB=1.4375, forceMergeDeletesPctAllowed=7.88214974082647, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.477010148973108
   [junit4]   2> 886654 WARN  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 886685 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 886685 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 886686 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 886686 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 886687 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1005444240, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 886687 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.SolrIndexSearcher Opening [Searcher@131f1b04[collection1_shard1_replica_n0] main]
   [junit4]   2> 886688 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 886688 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 886689 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 886690 INFO  (searcherExecutor-3145-thread-1-processing-n:127.0.0.1:41363_ x:collection1_shard1_replica_n0 s:shard1 c:collection1) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrCore [collection1_shard1_replica_n0] Registered new searcher Searcher@131f1b04[collection1_shard1_replica_n0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 886690 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572775282916458496
   [junit4]   2> 886694 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 886694 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 886694 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:41363/collection1_shard1_replica_n0/
   [junit4]   2> 886694 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 886694 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SyncStrategy http://127.0.0.1:41363/collection1_shard1_replica_n0/ has no replicas
   [junit4]   2> 886694 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 886697 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:41363/collection1_shard1_replica_n0/ shard1
   [junit4]   2> 886799 INFO  (zkCallback-1921-thread-1-processing-n:127.0.0.1:41363_) [n:127.0.0.1:41363_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 886799 INFO  (zkCallback-1915-thread-1-processing-n:127.0.0.1:36929_) [n:127.0.0.1:36929_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 886848 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 886849 INFO  (qtp87552865-9087) [n:127.0.0.1:41363_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n0&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1307
   [junit4]   2> 886851 INFO  (qtp87552865-9085) [n:127.0.0.1:41363_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:41363_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1312
   [junit4]   2> 886927 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/shard-3-001 of type NRT
   [junit4]   2> 886927 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 886928 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7449ad3c{/,null,AVAILABLE}
   [junit4]   2> 886928 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@36a4c247{HTTP/1.1,[http/1.1]}{127.0.0.1:36214}
   [junit4]   2> 886928 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.e.j.s.Server Started @889005ms
   [junit4]   2> 886928 INFO  (TEST-HttpPartitionTest.test-seed#[57BDCF540E9917CD]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/, hostPort=35393, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_57BDCF540E9917CD-001/shard-3-001/cores}
   [junit4]   2> 886929 ERROR (TEST-HttpPartitionTest.test-seed#[57BDCF54

[...truncated too long message...]

   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:204)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest.teardownClass(HdfsChaosMonkeySafeLeaderTest.java:47)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> ... 54 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_57BDCF540E9917CD-001
   [junit4]   2> Jul 13, 2017 3:44:56 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 34 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {rnd_b=PostingsFormat(name=Asserting), a_t=PostingsFormat(name=LuceneVarGapFixedInterval), a_i=PostingsFormat(name=Asserting), id=PostingsFormat(name=Asserting)}, docValues:{_version_=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=463, maxMBSortInHeap=6.128585903373934, sim=RandomSimilarity(queryNorm=true): {}, locale=es-CO, timezone=Africa/Bissau
   [junit4]   2> NOTE: Linux 4.4.0-83-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=2,free=165640360,total=529006592
   [junit4]   2> NOTE: All tests run in this JVM: [SmileWriterTest, TestMultiWordSynonyms, ZkNodePropsTest, UUIDFieldTest, TestPayloadCheckQParserPlugin, TestTolerantUpdateProcessorRandomCloud, TestWriterPerf, TestNestedDocsSort, TestCustomStream, TestJavabinTupleStreamParser, TestFieldCacheVsDocValues, DeleteShardTest, TestRangeQuery, TestHighlightDedupGrouping, TestSSLRandomization, OverseerStatusTest, TestStressUserVersions, OutputWriterTest, RecoveryAfterSoftCommitTest, TestConfigsApi, SolrCLIZkUtilsTest, CoreMergeIndexesAdminHandlerTest, TestSchemaNameResource, ClassificationUpdateProcessorTest, ResponseHeaderTest, SpellCheckCollatorTest, TestSolrCLIRunExample, TestConfigSetProperties, TestSolrQueryParser, PluginInfoTest, TestFaceting, TestInPlaceUpdatesDistrib, PropertiesRequestHandlerTest, FieldAnalysisRequestHandlerTest, TestSolrCoreSnapshots, TestSQLHandler, TolerantUpdateProcessorTest, DefaultValueUpdateProcessorTest, DistributedVersionInfoTest, TestNumericTokenStream, TestBlobHandler, TestLocalFSCloudBackupRestore, CurrencyFieldTypeTest, HdfsLockFactoryTest, AtomicUpdatesTest, BlockJoinFacetSimpleTest, SSLMigrationTest, TestSizeLimitedDistributedMap, TestBackupRepositoryFactory, TestDownShardTolerantSearch, HLLSerializationTest, VMParamsZkACLAndCredentialsProvidersTest, EnumFieldTest, DistributedSpellCheckComponentTest, TestSolrConfigHandlerConcurrent, TestManagedResource, TestReplicationHandler, BlobRepositoryCloudTest, TestImpersonationWithHadoopAuth, DistributedFacetPivotWhiteBoxTest, RequestHandlersTest, TestReloadAndDeleteDocs, TestXmlQParserPlugin, TestLRUStatsCache, OpenExchangeRatesOrgProviderTest, TestSchemaManager, NodeMutatorTest, TestLegacyNumericRangeQueryBuilder, TestRawTransformer, MetricUtilsTest, TestRealTimeGet, TestCollapseQParserPlugin, CdcrRequestHandlerTest, TestIndexingPerformance, HighlighterConfigTest, HttpPartitionTest, TestComponentsName, TestCloudManagedSchema, BadCopyFieldTest, TestRuleBasedAuthorizationPlugin, DebugComponentTest, TestDistributedSearch, TestLeaderElectionWithEmptyReplica, HdfsSyncSliceTest, SpellingQueryConverterTest, ReplicationFactorTest, TestOmitPositions, TestSolrCoreParser, TestSolr4Spatial, RankQueryTest, TestSimpleQParserPlugin, LeaderFailureAfterFreshStartTest, ConnectionReuseTest, TestReplicationHandlerBackup, SolrPluginUtilsTest, TestSolrConfigHandler, SolrIndexConfigTest, TestJsonRequest, CursorMarkTest, StatsReloadRaceTest, TestLegacyFieldReuse, TestMergePolicyConfig, TestFreeTextSuggestions, TestSolrDeletionPolicy1, ExitableDirectoryReaderTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, RAMDirectoryFactoryTest, TestUseDocValuesAsStored2, TestSchemaVersionResource, WrapperMergePolicyFactoryTest, FieldMutatingUpdateProcessorTest, TestConfig, V2StandaloneTest, MBeansHandlerTest, ImplicitSnitchTest, MetricsConfigTest, MigrateRouteKeyTest, LeaderElectionIntegrationTest, BasicZkTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, TestRandomFaceting, ZkSolrClientTest, TestRandomDVFaceting, TestReload, SolrCmdDistributorTest, QueryElevationComponentTest, PeerSyncTest, TestLazyCores, DistributedQueryElevationComponentTest, SolrIndexSplitterTest, SimplePostToolTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, TestExtendedDismaxParser, SolrRequestParserTest, TestFoldingMultitermQuery, SuggesterTest, TestStressLucene, SpatialFilterTest, SuggesterWFSTTest, TestCSVLoader, PolyFieldTest, WordBreakSolrSpellCheckerTest, SolrCoreCheckLockOnStartupTest, TestPseudoReturnFields, TestUpdate, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, SortByFunctionTest, TestRemoteStreaming, DocValuesMultiTest, SolrInfoBeanTest, IndexBasedSpellCheckerTest, DisMaxRequestHandlerTest, TestQueryUtils, DirectSolrSpellCheckerTest, TestReversedWildcardFilterFactory, DocumentAnalysisRequestHandlerTest, TestQueryTypes, XmlUpdateRequestHandlerTest, TermVectorComponentTest, MoreLikeThisHandlerTest, RequiredFieldsTest, LoggingHandlerTest, RegexBoostProcessorTest, IndexSchemaTest, TestCollationField, ReturnFieldsTest, TestCSVResponseWriter, QueryParsingTest, TestAnalyzedSuggestions, JsonLoaderTest, BinaryUpdateRequestHandlerTest, TestPHPSerializedResponseWriter, PingRequestHandlerTest, SampleTest, TestDocSet, TestBinaryField, NumericFieldsTest, MinimalSchemaTest, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestSolrCoreProperties, TestLuceneMatchVersion, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCharFilters, SynonymTokenizerTest, EchoParamsTest, TestSweetSpotSimilarityFactory, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, TimeZoneUtilsTest, TestFastOutputStream, URLClassifyProcessorTest, ChaosMonkeyNothingIsSafeTest, CdcrVersionReplicationTest, ChaosMonkeySafeLeaderWithPullReplicasTest, CleanupOldIndexTest, CloudExitableDirectoryReaderTest, CollectionStateFormat2Test, CollectionTooManyReplicasTest, CollectionsAPISolrJTest, ConfigSetsAPITest, ConnectionManagerTest, CreateCollectionCleanupTest, CustomCollectionTest, DeleteStatusTest, DistribJoinFromCollectionTest, ForceLeaderTest, MoveReplicaTest, RestartWhileUpdatingTest, TestCloudPseudoReturnFields, TestClusterProperties, TestConfigSetsAPI, TestOnReconnectListenerSupport, TestRandomFlRTGCloud, TestStressLiveNodes, AutoScalingHandlerTest, TestPolicyCloud, HdfsBasicDistributedZk2Test, HdfsChaosMonkeySafeLeaderTest]
   [junit4] Completed [568/728 (4!)] on J0 in 83.83s, 1 test, 1 failure <<< FAILURES!

[...truncated 43820 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 9 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/9/

6 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([ABCE03BB53DB72B5:239A3C61FD271F4D]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test(ChaosMonkeySafeLeaderWithPullReplicasTest.java:174)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
6 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:     1) Thread[id=90000, name=StoppableCommitThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)    2) Thread[id=90037, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    3) Thread[id=90036, name=StoppableCommitThread-SendThread(127.0.0.1:37765), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    4) Thread[id=90038, name=zkCallback-3567-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=90087, name=zkCallback-3567-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=90086, name=zkCallback-3567-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 6 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:
   1) Thread[id=90000, name=StoppableCommitThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)
   2) Thread[id=90037, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   3) Thread[id=90036, name=StoppableCommitThread-SendThread(127.0.0.1:37765), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   4) Thread[id=90038, name=zkCallback-3567-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=90087, name=zkCallback-3567-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=90086, name=zkCallback-3567-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([ABCE03BB53DB72B5]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=90036, name=StoppableCommitThread-SendThread(127.0.0.1:37765), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    2) Thread[id=90038, name=zkCallback-3567-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=90087, name=zkCallback-3567-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=90086, name=zkCallback-3567-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=90036, name=StoppableCommitThread-SendThread(127.0.0.1:37765), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   2) Thread[id=90038, name=zkCallback-3567-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=90087, name=zkCallback-3567-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=90086, name=zkCallback-3567-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([ABCE03BB53DB72B5]:0)


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at __randomizedtesting.SeedInfo.seed([ABCE03BB53DB72B5:239A3C61FD271F4D]:0)
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649)
        at java.lang.StringBuilder.append(StringBuilder.java:202)
        at org.apache.http.client.utils.URLEncodedUtils.urlEncode(URLEncodedUtils.java:522)
        at org.apache.http.client.utils.URLEncodedUtils.encodeFormFields(URLEncodedUtils.java:638)
        at org.apache.http.client.utils.URLEncodedUtils.format(URLEncodedUtils.java:390)
        at org.apache.http.client.utils.URLEncodedUtils.format(URLEncodedUtils.java:369)
        at org.apache.http.client.entity.UrlEncodedFormEntity.<init>(UrlEncodedFormEntity.java:77)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:447)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:974)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:990)
        at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:228)
        at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:167)
        at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:668)
        at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:
Timeout waiting for all live and active

Stack Trace:
java.lang.AssertionError: Timeout waiting for all live and active
        at __randomizedtesting.SeedInfo.seed([ABCE03BB53DB72B5:239A3C61FD271F4D]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:246)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:136)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1500018646551,generation=2,filelist=[_w3.cfe, _w3.cfs, _w3.si, _x5.fdt, _x5.fdx, _x5.fnm, _x5.nvd, _x5.nvm, _x5.si, _x5_Memory_0.ram, _x9.fdt, _x9.fdx, _x9.fnm, _x9.nvd, _x9.nvm, _x9.si, _x9_Memory_0.ram, _xb.cfe, _xb.cfs, _xb.si, _xc.fdt, _xc.fdx, _xc.fnm, _xc.nvd, _xc.nvm, _xc.si, _xc_Memory_0.ram, _xd.fdt, _xd.fdx, _xd.fnm, _xd.nvd, _xd.nvm, _xd.si, _xd_Memory_0.ram, _xe.fdt, _xe.fdx, _xe.fnm, _xe.nvd, _xe.nvm, _xe.si, _xe_Memory_0.ram, segments_2]}]> but was:<[{indexVersion=1500018646551,generation=2,filelist=[_w3.cfe, _w3.cfs, _w3.si, _x5.fdt, _x5.fdx, _x5.fnm, _x5.nvd, _x5.nvm, _x5.si, _x5_Memory_0.ram, _x9.fdt, _x9.fdx, _x9.fnm, _x9.nvd, _x9.nvm, _x9.si, _x9_Memory_0.ram, _xb.cfe, _xb.cfs, _xb.si, _xc.fdt, _xc.fdx, _xc.fnm, _xc.nvd, _xc.nvm, _xc.si, _xc_Memory_0.ram, _xd.fdt, _xd.fdx, _xd.fnm, _xd.nvd, _xd.nvm, _xd.si, _xd_Memory_0.ram, _xe.fdt, _xe.fdx, _xe.fnm, _xe.nvd, _xe.nvm, _xe.si, _xe_Memory_0.ram, segments_2]}, {indexVersion=1500018646551,generation=3,filelist=[_x4.cfe, _x4.cfs, _x4.si, _x5.fdt, _x5.fdx, _x5.fnm, _x5.nvd, _x5.nvm, _x5.si, _x5_Memory_0.ram, _x9.fdt, _x9.fdx, _x9.fnm, _x9.nvd, _x9.nvm, _x9.si, _x9_Memory_0.ram, _xb.cfe, _xb.cfs, _xb.si, _xc.fdt, _xc.fdx, _xc.fnm, _xc.nvd, _xc.nvm, _xc.si, _xc_Memory_0.ram, _xd.fdt, _xd.fdx, _xd.fnm, _xd.nvd, _xd.nvm, _xd.si, _xd_Memory_0.ram, _xe.fdt, _xe.fdx, _xe.fnm, _xe.nvd, _xe.nvm, _xe.si, _xe_Memory_0.ram, segments_3]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1500018646551,generation=2,filelist=[_w3.cfe, _w3.cfs, _w3.si, _x5.fdt, _x5.fdx, _x5.fnm, _x5.nvd, _x5.nvm, _x5.si, _x5_Memory_0.ram, _x9.fdt, _x9.fdx, _x9.fnm, _x9.nvd, _x9.nvm, _x9.si, _x9_Memory_0.ram, _xb.cfe, _xb.cfs, _xb.si, _xc.fdt, _xc.fdx, _xc.fnm, _xc.nvd, _xc.nvm, _xc.si, _xc_Memory_0.ram, _xd.fdt, _xd.fdx, _xd.fnm, _xd.nvd, _xd.nvm, _xd.si, _xd_Memory_0.ram, _xe.fdt, _xe.fdx, _xe.fnm, _xe.nvd, _xe.nvm, _xe.si, _xe_Memory_0.ram, segments_2]}]> but was:<[{indexVersion=1500018646551,generation=2,filelist=[_w3.cfe, _w3.cfs, _w3.si, _x5.fdt, _x5.fdx, _x5.fnm, _x5.nvd, _x5.nvm, _x5.si, _x5_Memory_0.ram, _x9.fdt, _x9.fdx, _x9.fnm, _x9.nvd, _x9.nvm, _x9.si, _x9_Memory_0.ram, _xb.cfe, _xb.cfs, _xb.si, _xc.fdt, _xc.fdx, _xc.fnm, _xc.nvd, _xc.nvm, _xc.si, _xc_Memory_0.ram, _xd.fdt, _xd.fdx, _xd.fnm, _xd.nvd, _xd.nvm, _xd.si, _xd_Memory_0.ram, _xe.fdt, _xe.fdx, _xe.fnm, _xe.nvd, _xe.nvm, _xe.si, _xe_Memory_0.ram, segments_2]}, {indexVersion=1500018646551,generation=3,filelist=[_x4.cfe, _x4.cfs, _x4.si, _x5.fdt, _x5.fdx, _x5.fnm, _x5.nvd, _x5.nvm, _x5.si, _x5_Memory_0.ram, _x9.fdt, _x9.fdx, _x9.fnm, _x9.nvd, _x9.nvm, _x9.si, _x9_Memory_0.ram, _xb.cfe, _xb.cfs, _xb.si, _xc.fdt, _xc.fdx, _xc.fnm, _xc.nvd, _xc.nvm, _xc.si, _xc_Memory_0.ram, _xd.fdt, _xd.fdx, _xd.fnm, _xd.nvd, _xd.nvm, _xd.si, _xd_Memory_0.ram, _xe.fdt, _xe.fdx, _xe.fnm, _xe.nvd, _xe.nvm, _xe.si, _xe_Memory_0.ram, segments_3]}]>
        at __randomizedtesting.SeedInfo.seed([ABCE03BB53DB72B5:8E19188B23937CB6]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1277)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12426 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/init-core-data-001
   [junit4]   2> 2896291 WARN  (SUITE-TestReplicationHandler-seed#[ABCE03BB53DB72B5]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 2896292 INFO  (SUITE-TestReplicationHandler-seed#[ABCE03BB53DB72B5]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2896293 INFO  (SUITE-TestReplicationHandler-seed#[ABCE03BB53DB72B5]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 2896294 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestRepeater
   [junit4]   2> 2896294 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-001/collection1
   [junit4]   2> 2896296 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2896297 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2306f184{/solr,null,AVAILABLE}
   [junit4]   2> 2896297 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1065cb39{HTTP/1.1,[http/1.1]}{127.0.0.1:41611}
   [junit4]   2> 2896298 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.Server Started @2905864ms
   [junit4]   2> 2896298 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=41611}
   [junit4]   2> 2896298 ERROR (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2896298 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 2896298 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 2896298 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 2896298 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-14T07:46:53.862Z
   [junit4]   2> 2896298 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-001/solr.xml
   [junit4]   2> 2896302 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2896337 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2896340 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2896340 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2896341 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-001/.
   [junit4]   2> 2896341 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2896350 INFO  (coreLoadExecutor-7789-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 2896386 INFO  (coreLoadExecutor-7789-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2896396 INFO  (coreLoadExecutor-7789-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 2896396 INFO  (coreLoadExecutor-7789-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 2896396 INFO  (coreLoadExecutor-7789-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2896396 INFO  (coreLoadExecutor-7789-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2896396 INFO  (coreLoadExecutor-7789-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-001/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 2896397 INFO  (coreLoadExecutor-7789-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6226604577629139]
   [junit4]   2> 2896479 INFO  (coreLoadExecutor-7789-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2896479 INFO  (coreLoadExecutor-7789-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2896480 INFO  (coreLoadExecutor-7789-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@188a3d90
   [junit4]   2> 2896480 INFO  (coreLoadExecutor-7789-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@61b1c677[collection1] main]
   [junit4]   2> 2896480 INFO  (coreLoadExecutor-7789-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-001/collection1/conf
   [junit4]   2> 2896489 INFO  (coreLoadExecutor-7789-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 2896489 INFO  (coreLoadExecutor-7789-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2896490 INFO  (searcherExecutor-7790-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@61b1c677[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2896886 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/collection1
   [junit4]   2> 2896888 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2896889 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@79653f7e{/solr,null,AVAILABLE}
   [junit4]   2> 2896889 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5f8fe966{HTTP/1.1,[http/1.1]}{127.0.0.1:41753}
   [junit4]   2> 2896889 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.Server Started @2906456ms
   [junit4]   2> 2896889 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=41753}
   [junit4]   2> 2896890 ERROR (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2896890 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 2896890 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 2896890 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 2896890 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-14T07:46:54.454Z
   [junit4]   2> 2896891 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/solr.xml
   [junit4]   2> 2896940 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2896982 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2896986 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2896986 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2896988 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/.
   [junit4]   2> 2896988 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2896996 INFO  (coreLoadExecutor-7799-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 2897028 INFO  (coreLoadExecutor-7799-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2897029 INFO  (coreLoadExecutor-7799-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 2897030 INFO  (coreLoadExecutor-7799-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 2897030 INFO  (coreLoadExecutor-7799-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2897030 INFO  (coreLoadExecutor-7799-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2897030 INFO  (coreLoadExecutor-7799-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 2897031 INFO  (coreLoadExecutor-7799-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6226604577629139]
   [junit4]   2> 2897093 INFO  (coreLoadExecutor-7799-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2897093 INFO  (coreLoadExecutor-7799-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2897094 INFO  (coreLoadExecutor-7799-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@1e2a6d93
   [junit4]   2> 2897094 INFO  (coreLoadExecutor-7799-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3c895466[collection1] main]
   [junit4]   2> 2897094 INFO  (coreLoadExecutor-7799-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/collection1/conf
   [junit4]   2> 2897099 INFO  (coreLoadExecutor-7799-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 2897100 INFO  (searcherExecutor-7800-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3c895466[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2897100 INFO  (qtp1630951948-121935) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 2897100 INFO  (indexFetcher-7804-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 2897101 INFO  (indexFetcher-7804-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 2897101 INFO  (indexFetcher-7804-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 2897101 INFO  (indexFetcher-7804-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 2897554 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@5f8fe966{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2897554 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1918352296
   [junit4]   2> 2897554 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2897555 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2897555 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2897555 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2897556 INFO  (coreCloseExecutor-7805-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@658fb64f
   [junit4]   2> 2897557 INFO  (coreCloseExecutor-7805-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1703917135
   [junit4]   2> 2897558 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@79653f7e{/solr,null,UNAVAILABLE}
   [junit4]   2> 2897559 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2897559 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@78e741bd{/solr,null,AVAILABLE}
   [junit4]   2> 2897560 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4a0127ff{HTTP/1.1,[http/1.1]}{127.0.0.1:35054}
   [junit4]   2> 2897560 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.Server Started @2907126ms
   [junit4]   2> 2897560 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=35054}
   [junit4]   2> 2897560 ERROR (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2897560 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 2897560 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 2897560 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 2897560 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-14T07:46:55.124Z
   [junit4]   2> 2897561 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/solr.xml
   [junit4]   2> 2897565 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2897600 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2897604 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2897604 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2897605 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/.
   [junit4]   2> 2897605 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2897634 INFO  (coreLoadExecutor-7812-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 2897670 INFO  (coreLoadExecutor-7812-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2897671 INFO  (coreLoadExecutor-7812-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 2897671 INFO  (coreLoadExecutor-7812-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 2897672 INFO  (coreLoadExecutor-7812-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2897672 INFO  (coreLoadExecutor-7812-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2897684 INFO  (coreLoadExecutor-7812-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 2897685 INFO  (coreLoadExecutor-7812-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6226604577629139]
   [junit4]   2> 2897814 INFO  (coreLoadExecutor-7812-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2897814 INFO  (coreLoadExecutor-7812-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2897827 INFO  (coreLoadExecutor-7812-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@6deb6047
   [junit4]   2> 2897827 INFO  (coreLoadExecutor-7812-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@f8f0809[collection1] main]
   [junit4]   2> 2897827 INFO  (coreLoadExecutor-7812-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/collection1/conf
   [junit4]   2> 2897828 INFO  (coreLoadExecutor-7812-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2897828 INFO  (searcherExecutor-7813-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@f8f0809[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2898157 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-003/collection1
   [junit4]   2> 2898159 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2898194 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@227ab761{/solr,null,AVAILABLE}
   [junit4]   2> 2898248 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3e8329f0{HTTP/1.1,[http/1.1]}{127.0.0.1:44104}
   [junit4]   2> 2898248 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.e.j.s.Server Started @2907815ms
   [junit4]   2> 2898248 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-003/collection1/data, hostContext=/solr, hostPort=44104}
   [junit4]   2> 2898248 ERROR (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2898249 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 2898249 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 2898249 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 2898249 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-14T07:46:55.813Z
   [junit4]   2> 2898249 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-003/solr.xml
   [junit4]   2> 2898253 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2898313 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2898317 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2898317 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2898323 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-003/.
   [junit4]   2> 2898323 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2898333 INFO  (coreLoadExecutor-7822-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 2898374 INFO  (coreLoadExecutor-7822-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2898392 INFO  (coreLoadExecutor-7822-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 2898392 INFO  (coreLoadExecutor-7822-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-003/./collection1, trusted=true
   [junit4]   2> 2898392 INFO  (coreLoadExecutor-7822-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2898392 INFO  (coreLoadExecutor-7822-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2898392 INFO  (coreLoadExecutor-7822-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-003/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-003/./collection1/data/]
   [junit4]   2> 2898393 INFO  (coreLoadExecutor-7822-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6226604577629139]
   [junit4]   2> 2898435 INFO  (coreLoadExecutor-7822-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2898435 INFO  (coreLoadExecutor-7822-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2898436 INFO  (coreLoadExecutor-7822-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@2d34a4f9
   [junit4]   2> 2898436 INFO  (coreLoadExecutor-7822-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@58c5d3c0[collection1] main]
   [junit4]   2> 2898436 INFO  (coreLoadExecutor-7822-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-003/collection1/conf
   [junit4]   2> 2898437 WARN  (coreLoadExecutor-7822-thread-1) [    x:collection1] o.a.s.h.IndexFetcher 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 2898439 INFO  (coreLoadExecutor-7822-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler  No value set for 'pollInterval'. Timer Task not started.
   [junit4]   2> 2898439 INFO  (coreLoadExecutor-7822-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml
   [junit4]   2> 2898439 INFO  (coreLoadExecutor-7822-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2898439 INFO  (searcherExecutor-7823-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@58c5d3c0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2898904 INFO  (qtp1630951948-121939) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 0
   [junit4]   2> 2898904 INFO  (qtp1630951948-121932) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 0
   [junit4]   2> 2898905 INFO  (qtp1630951948-121935) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2]} 0 0
   [junit4]   2> 2898905 INFO  (qtp1630951948-121936) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2898905 INFO  (qtp1630951948-121936) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@158f28cf commitCommandVersion:0
   [junit4]   2> 2898907 INFO  (qtp1630951948-121936) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@31b2070a[collection1] main]
   [junit4]   2> 2898907 INFO  (qtp1630951948-121936) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2898908 INFO  (searcherExecutor-7790-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@31b2070a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):C3)))}
   [junit4]   2> 2898908 INFO  (qtp1630951948-121936) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 3
   [junit4]   2> 2898910 WARN  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 2898911 INFO  (qtp1630951948-121933) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 2898911 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 2
   [junit4]   2> 2898911 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Master's version: 1500018416469
   [junit4]   2> 2898911 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 2898911 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 2898911 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 2898913 INFO  (qtp1630951948-121939) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=1
   [junit4]   2> 2898913 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Number of files in latest index in master: 8
   [junit4]   2> 2898914 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=38, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2898915 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 2898915 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting download (fullCopy=false) to MockDirectoryWrapper(RAMDirectory@742ac711 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@349b16f4)
   [junit4]   2> 2898915 INFO  (qtp1630951948-121932) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.si&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2898932 INFO  (qtp1630951948-121935) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.nvd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2898933 INFO  (qtp1630951948-121936) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fdx&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2898933 INFO  (qtp1630951948-121937) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Memory_0.ram&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2898934 INFO  (qtp1630951948-121933) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fdt&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2898934 INFO  (qtp1630951948-121939) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fnm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2898936 INFO  (qtp1630951948-121932) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.nvm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2898937 INFO  (qtp1630951948-121935) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=segments_2&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2898937 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=false,bytesDownloaded=1380) : 0 secs (null bytes/sec) to MockDirectoryWrapper(RAMDirectory@742ac711 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@349b16f4)
   [junit4]   2> 2898940 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=1.57421875, floorSegmentMB=1.9482421875, forceMergeDeletesPctAllowed=8.568864096386566, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.39460049999639224
   [junit4]   2> 2898941 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 2898941 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@750ed32f[collection1] main]
   [junit4]   2> 2898942 INFO  (searcherExecutor-7823-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@750ed32f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):C3)))}
   [junit4]   2> 2898942 INFO  (qtp55242568-121991) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={wait=true&masterUrl=http://127.0.0.1:41611/solr/collection1/replication&command=fetchindex} status=0 QTime=33
   [junit4]   2> 2898944 INFO  (qtp55242568-121993) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=3 status=0 QTime=0
   [junit4]   2> 2898944 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.h.TestReplicationHandler Waited for 0ms and found 3 docs
   [junit4]   2> 2898945 WARN  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 2898946 INFO  (qtp55242568-121995) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 2898946 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 2
   [junit4]   2> 2898946 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Master's version: 1500018416469
   [junit4]   2> 2898946 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 2898946 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 2898946 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 2898946 INFO  (qtp55242568-121988) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=0
   [junit4]   2> 2898947 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Number of files in latest index in master: 8
   [junit4]   2> 2898948 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=38, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2898948 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 2898948 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting download (fullCopy=false) to MockDirectoryWrapper(RAMDirectory@7c783059 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@171da3ce)
   [junit4]   2> 2898949 INFO  (qtp55242568-121991) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.si&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2898949 INFO  (qtp55242568-121989) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.nvd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2898950 INFO  (qtp55242568-121993) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fdx&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2898951 INFO  (qtp55242568-121994) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Memory_0.ram&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2898951 INFO  (qtp55242568-121995) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fdt&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2898960 INFO  (qtp55242568-121988) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fnm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2898961 INFO  (qtp55242568-121991) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.nvm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2898961 INFO  (qtp55242568-121989) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=segments_2&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2898962 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=false,bytesDownloaded=1380) : 0 secs (null bytes/sec) to MockDirectoryWrapper(RAMDirectory@7c783059 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@171da3ce)
   [junit4]   2> 2898964 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=1.57421875, floorSegmentMB=1.9482421875, forceMergeDeletesPctAllowed=8.568864096386566, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.39460049999639224
   [junit4]   2> 2898965 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 2898965 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@68dff132[collection1] main]
   [junit4]   2> 2898966 INFO  (searcherExecutor-7813-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@68dff132[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):C3)))}
   [junit4]   2> 2898966 INFO  (qtp343049728-121974) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={wait=true&masterUrl=http://127.0.0.1:44104/solr/collection1/replication&command=fetchindex} status=0 QTime=21
   [junit4]   2> 2898973 INFO  (qtp343049728-121976) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=3 status=0 QTime=0
   [junit4]   2> 2898973 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] o.a.s.h.TestReplicationHandler Waited for 0ms and found 3 docs
   [junit4]   2> 2898973 INFO  (qtp1630951948-121938) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 2898974 INFO  (qtp1630951948-121937) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 2898975 WARN  (qtp55242568-121993) [    x:collection1] o.a.s.h.ReplicationHandler Exception while invoking 'details' method for replication on master
   [junit4]   2> org.apache.solr.client.solrj.SolrServerException: Server refused connection at: <a href="http://127.0.0.1:TEST_PORT/solr">http://127.0.0.1:TEST_PORT/solr
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:633)
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:252)
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
   [junit4]   2> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]   2> at org.apache.solr.handler.IndexFetcher.getDetails(IndexFetcher.java:1813)
   [junit4]   2> at org.apache.solr.handler.ReplicationHandler.getReplicationDetails(ReplicationHandler.java:952)
   [junit4]   2> at org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:330)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2473)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:462)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused (Connection refused)
   [junit4]   2> at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
   [junit4]   2> at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
   [junit4]   2> at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
   [junit4]   2> at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
   [junit4]   2> at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
   [junit4]   2> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
   [junit4]   2> at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
   [junit4]   2> at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
   [junit4]   2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:523)
   [junit4]   2> ... 36 more
   [junit4]   2> Caused by: java.net.ConnectException: Connection refused (Connection refused)
   [junit4]   2> at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]   2> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
   [junit4]   2> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
   [junit4]   2> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketIm

[...truncated too long message...]

cale=und -Dtests.timezone=Asia/Dubai -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J2 | ChaosMonkeySafeLeaderWithPullReplicasTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 6 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:
   [junit4]    >    1) Thread[id=90000, name=StoppableCommitThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)
   [junit4]    >    2) Thread[id=90037, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   [junit4]    >    3) Thread[id=90036, name=StoppableCommitThread-SendThread(127.0.0.1:37765), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   [junit4]    >    4) Thread[id=90038, name=zkCallback-3567-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    5) Thread[id=90087, name=zkCallback-3567-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    6) Thread[id=90086, name=zkCallback-3567-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([ABCE03BB53DB72B5]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=90036, name=StoppableCommitThread-SendThread(127.0.0.1:37765), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   [junit4]    >    2) Thread[id=90038, name=zkCallback-3567-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    3) Thread[id=90087, name=zkCallback-3567-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    4) Thread[id=90086, name=zkCallback-3567-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([ABCE03BB53DB72B5]:0)
   [junit4] Completed [502/728 (4!)] on J2 in 136.56s, 1 test, 1 failure, 2 errors <<< FAILURES!

[...truncated 685 lines...]
   [junit4] JVM J0: stdout was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J0-20170714_065827_8894968092063555578472.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/heapdumps/java_pid20311.hprof ...
   [junit4] Heap dump file created [555493137 bytes in 8.784 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 7504 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:818: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:762: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid20311.hprof

Total time: 467 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 10 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/10/

3 tests failed.
FAILED:  org.apache.lucene.spatial.DistanceStrategyTest.testRecipScore {strategy=serialized}

Error Message:


Stack Trace:
java.lang.NullPointerException
        at __randomizedtesting.SeedInfo.seed([B1FC2FB143140AF1:29D25F48109F000A]:0)
        at org.apache.lucene.spatial.serialized.SerializedDVStrategy$ShapeDocValueSource$1.advanceExact(SerializedDVStrategy.java:186)
        at org.apache.lucene.spatial.util.DistanceToShapeValueSource$1.advanceExact(DistanceToShapeValueSource.java:73)
        at org.apache.lucene.search.DoubleValues$1.advanceExact(DoubleValues.java:53)
        at org.apache.lucene.spatial.util.ReciprocalDoubleValuesSource$1.advanceExact(ReciprocalDoubleValuesSource.java:57)
        at org.apache.lucene.spatial.StrategyTestCase.checkValueSource(StrategyTestCase.java:219)
        at org.apache.lucene.spatial.DistanceStrategyTest.testRecipScore(DistanceStrategyTest.java:104)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.update.AutoCommitTest.testMaxDocs

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
        at __randomizedtesting.SeedInfo.seed([793723D4943B9E3D:C0B6F50BB8D19AB7]:0)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:878)
        at org.apache.solr.update.AutoCommitTest.testMaxDocs(AutoCommitTest.java:225)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
        xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

        request was:q=id:14&qt=&start=0&rows=20&version=2.2
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:871)
        ... 40 more


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:
Timeout waiting for all live and active

Stack Trace:
java.lang.AssertionError: Timeout waiting for all live and active
        at __randomizedtesting.SeedInfo.seed([793723D4943B9E3D:F1631C0E3AC7F3C5]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:246)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:136)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 9261 lines...]
   [junit4] Suite: org.apache.lucene.spatial.DistanceStrategyTest
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DistanceStrategyTest -Dtests.method=testRecipScore -Dtests.seed=B1FC2FB143140AF1 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=ar-AE -Dtests.timezone=Pacific/Chatham -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.03s J0 | DistanceStrategyTest.testRecipScore {strategy=serialized} <<<
   [junit4]    > Throwable #1: java.lang.NullPointerException
   [junit4]    > at __randomizedtesting.SeedInfo.seed([B1FC2FB143140AF1:29D25F48109F000A]:0)
   [junit4]    > at org.apache.lucene.spatial.serialized.SerializedDVStrategy$ShapeDocValueSource$1.advanceExact(SerializedDVStrategy.java:186)
   [junit4]    > at org.apache.lucene.spatial.util.DistanceToShapeValueSource$1.advanceExact(DistanceToShapeValueSource.java:73)
   [junit4]    > at org.apache.lucene.search.DoubleValues$1.advanceExact(DoubleValues.java:53)
   [junit4]    > at org.apache.lucene.spatial.util.ReciprocalDoubleValuesSource$1.advanceExact(ReciprocalDoubleValuesSource.java:57)
   [junit4]    > at org.apache.lucene.spatial.StrategyTestCase.checkValueSource(StrategyTestCase.java:219)
   [junit4]    > at org.apache.lucene.spatial.DistanceStrategyTest.testRecipScore(DistanceStrategyTest.java:104)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {recursive_packedquad=PostingsFormat(name=MockRandom), recursive_quad=PostingsFormat(name=MockRandom), bbox__xdl=BlockTreeOrds(blocksize=128), termquery_geohash=PostingsFormat(name=MockRandom), id=BlockTreeOrds(blocksize=128)}, docValues:{bbox__maxX=DocValuesFormat(name=Direct), bbox__maxY=DocValuesFormat(name=Lucene70), bbox__minX=DocValuesFormat(name=Asserting), bbox__minY=DocValuesFormat(name=Memory), pointvector__y=DocValuesFormat(name=Lucene70), pointvector__x=DocValuesFormat(name=Direct), serialized=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=1913, maxMBSortInHeap=5.545002168373545, sim=RandomSimilarity(queryNorm=true): {}, locale=ar-AE, timezone=Pacific/Chatham
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=291830768,total=324534272
   [junit4]   2> NOTE: All tests run in this JVM: [SpatialPrefixTreeTest, NumberRangeFacetsTest, TestTermQueryPrefixGridStrategy, TestRecursivePrefixTreeStrategy, Geo3dShapeSphereModelRectRelationTest, DistanceStrategyTest]
   [junit4] Completed [12/23 (1!)] on J0 in 0.69s, 12 tests, 1 error <<< FAILURES!

[...truncated 2297 lines...]
   [junit4] Suite: org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_793723D4943B9E3D-001/init-core-data-001
   [junit4]   2> 548632 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[793723D4943B9E3D]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=38 numCloses=38
   [junit4]   2> 548632 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[793723D4943B9E3D]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 548634 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[793723D4943B9E3D]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 548634 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[793723D4943B9E3D]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /ov/cn
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 548849 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[793723D4943B9E3D]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 548855 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[793723D4943B9E3D]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 548857 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[793723D4943B9E3D]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 548870 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[793723D4943B9E3D]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_56321_hdfs____.206o6d/webapp
   [junit4]   2> 549299 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[793723D4943B9E3D]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:56321
   [junit4]   2> 549422 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[793723D4943B9E3D]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 549423 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[793723D4943B9E3D]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 549436 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[793723D4943B9E3D]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_59355_datanode____.vqlzu2/webapp
   [junit4]   2> 549878 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[793723D4943B9E3D]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:59355
   [junit4]   2> 549915 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[793723D4943B9E3D]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 549916 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[793723D4943B9E3D]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 549934 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[793723D4943B9E3D]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_54759_datanode____jrk8hv/webapp
   [junit4]   2> 550164 INFO  (IPC Server handler 4 on 49971) [    ] BlockStateChange BLOCK* processReport: from storage DS-e9ea3dbc-9113-4645-b893-130c3730ec1a node DatanodeRegistration(127.0.0.1:50603, datanodeUuid=527ac2ae-b401-4d84-86e3-6f935dee3843, infoPort=54653, infoSecurePort=0, ipcPort=36188, storageInfo=lv=-56;cid=testClusterID;nsid=451575161;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 550164 INFO  (IPC Server handler 4 on 49971) [    ] BlockStateChange BLOCK* processReport: from storage DS-ada032ef-beb3-4321-b044-13278d645633 node DatanodeRegistration(127.0.0.1:50603, datanodeUuid=527ac2ae-b401-4d84-86e3-6f935dee3843, infoPort=54653, infoSecurePort=0, ipcPort=36188, storageInfo=lv=-56;cid=testClusterID;nsid=451575161;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 550420 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[793723D4943B9E3D]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:54759
   [junit4]   2> 550918 INFO  (IPC Server handler 9 on 49971) [    ] BlockStateChange BLOCK* processReport: from storage DS-7b441025-265d-40cb-ac06-5e2d91c89e3b node DatanodeRegistration(127.0.0.1:60224, datanodeUuid=48e239a8-7fb1-4b7a-bad7-3965b9821989, infoPort=45761, infoSecurePort=0, ipcPort=36307, storageInfo=lv=-56;cid=testClusterID;nsid=451575161;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 550918 INFO  (IPC Server handler 9 on 49971) [    ] BlockStateChange BLOCK* processReport: from storage DS-4f0f9719-f907-44eb-a680-fe795758da6b node DatanodeRegistration(127.0.0.1:60224, datanodeUuid=48e239a8-7fb1-4b7a-bad7-3965b9821989, infoPort=45761, infoSecurePort=0, ipcPort=36307, storageInfo=lv=-56;cid=testClusterID;nsid=451575161;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 551053 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[793723D4943B9E3D]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 551060 INFO  (Thread-2097) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 551060 INFO  (Thread-2097) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 551075 ERROR (Thread-2097) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 551160 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[793723D4943B9E3D]) [    ] o.a.s.c.ZkTestServer start zk server on port:52694
   [junit4]   2> 551179 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[793723D4943B9E3D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 551181 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[793723D4943B9E3D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 551183 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[793723D4943B9E3D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 551185 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[793723D4943B9E3D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 551187 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[793723D4943B9E3D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 551189 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[793723D4943B9E3D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 551191 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[793723D4943B9E3D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 551192 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[793723D4943B9E3D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 551194 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[793723D4943B9E3D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 551196 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[793723D4943B9E3D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 551198 INFO  (TES