[JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 15 - Unstable

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 15 - Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.7/15/

4 tests failed.
FAILED:  org.apache.solr.cloud.MigrateRouteKeyTest.multipleShardMigrateTest

Error Message:
DocCount on target collection does not match expected:<8> but was:<9>

Stack Trace:
java.lang.AssertionError: DocCount on target collection does not match expected:<8> but was:<9>
        at __randomizedtesting.SeedInfo.seed([565588B2122F9CD2:4A4592C833EF2C33]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:834)
        at org.junit.Assert.assertEquals(Assert.java:645)
        at org.apache.solr.cloud.MigrateRouteKeyTest.multipleShardMigrateTest(MigrateRouteKeyTest.java:163)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.api.collections.ShardSplitTest.testSplitWithChaosMonkey

Error Message:
Address already in use

Stack Trace:
java.net.BindException: Address already in use
        at __randomizedtesting.SeedInfo.seed([565588B2122F9CD2:DD725B6353293756]:0)
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:342)
        at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:308)
        at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
        at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.server.Server.doStart(Server.java:394)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:510)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:449)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:417)
        at org.apache.solr.cloud.api.collections.ShardSplitTest.testSplitWithChaosMonkey(ShardSplitTest.java:499)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1075)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1047)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.api.collections.ShardSplitTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.api.collections.ShardSplitTest:     1) Thread[id=21370, name=qtp905665526-21370, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=21364, name=qtp905665526-21364, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=21363, name=qtp905665526-21363, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at org.eclipse.jetty.io.ManagedSelector$$Lambda$33/519868693.run(Unknown Source)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=21492, name=Scheduler-834984337, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=21367, name=qtp905665526-21367, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=21371, name=Scheduler-911839782, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=21366, name=qtp905665526-21366-acceptor-0@2515fa28-ServerConnector@51dcdfa0{HTTP/1.1,[http/1.1]}{127.0.0.1:43127}, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.lang.Thread.run(Thread.java:748)    8) Thread[id=21365, name=qtp905665526-21365, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at org.eclipse.jetty.io.ManagedSelector$$Lambda$33/519868693.run(Unknown Source)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.lang.Thread.run(Thread.java:748)    9) Thread[id=21368, name=qtp905665526-21368, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)         at java.lang.Thread.run(Thread.java:748)   10) Thread[id=21369, name=qtp905665526-21369, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.api.collections.ShardSplitTest:
   1) Thread[id=21370, name=qtp905665526-21370, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=21364, name=qtp905665526-21364, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=21363, name=qtp905665526-21363, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$33/519868693.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=21492, name=Scheduler-834984337, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=21367, name=qtp905665526-21367, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=21371, name=Scheduler-911839782, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=21366, name=qtp905665526-21366-acceptor-0@2515fa28-ServerConnector@51dcdfa0{HTTP/1.1,[http/1.1]}{127.0.0.1:43127}, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=21365, name=qtp905665526-21365, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$33/519868693.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)
   9) Thread[id=21368, name=qtp905665526-21368, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
        at java.lang.Thread.run(Thread.java:748)
  10) Thread[id=21369, name=qtp905665526-21369, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([565588B2122F9CD2]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.api.collections.ShardSplitTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=21371, name=Scheduler-911839782, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=21365, name=qtp905665526-21365, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at org.eclipse.jetty.io.ManagedSelector$$Lambda$33/519868693.run(Unknown Source)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=21363, name=qtp905665526-21363, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at org.eclipse.jetty.io.ManagedSelector$$Lambda$33/519868693.run(Unknown Source)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=21492, name=Scheduler-834984337, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=21371, name=Scheduler-911839782, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=21365, name=qtp905665526-21365, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$33/519868693.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=21363, name=qtp905665526-21363, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$33/519868693.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=21492, name=Scheduler-834984337, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([565588B2122F9CD2]:0)




Build Log:
[...truncated 13335 lines...]
   [junit4] Suite: org.apache.solr.cloud.MigrateRouteKeyTest
   [junit4]   2> 258274 INFO  (SUITE-MigrateRouteKeyTest-seed#[565588B2122F9CD2]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MigrateRouteKeyTest_565588B2122F9CD2-001/init-core-data-001
   [junit4]   2> 258275 WARN  (SUITE-MigrateRouteKeyTest-seed#[565588B2122F9CD2]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2514 numCloses=2514
   [junit4]   2> 258275 INFO  (SUITE-MigrateRouteKeyTest-seed#[565588B2122F9CD2]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 258277 INFO  (SUITE-MigrateRouteKeyTest-seed#[565588B2122F9CD2]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 258291 INFO  (SUITE-MigrateRouteKeyTest-seed#[565588B2122F9CD2]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MigrateRouteKeyTest_565588B2122F9CD2-001/tempDir-001
   [junit4]   2> 258292 INFO  (SUITE-MigrateRouteKeyTest-seed#[565588B2122F9CD2]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 258366 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 258366 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 258466 INFO  (SUITE-MigrateRouteKeyTest-seed#[565588B2122F9CD2]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:38092
   [junit4]   2> 258466 INFO  (SUITE-MigrateRouteKeyTest-seed#[565588B2122F9CD2]-worker) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:38092
   [junit4]   2> 258466 INFO  (SUITE-MigrateRouteKeyTest-seed#[565588B2122F9CD2]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 38092
   [junit4]   2> 258563 INFO  (zkConnectionManagerCallback-111-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 258583 INFO  (zkConnectionManagerCallback-113-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 258601 INFO  (zkConnectionManagerCallback-115-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 258658 WARN  (jetty-launcher-116-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 258693 WARN  (jetty-launcher-116-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 258694 INFO  (jetty-launcher-116-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 258694 INFO  (jetty-launcher-116-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 258694 INFO  (jetty-launcher-116-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 258707 INFO  (jetty-launcher-116-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 258707 INFO  (jetty-launcher-116-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 258707 INFO  (jetty-launcher-116-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 258730 INFO  (jetty-launcher-116-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 258731 INFO  (jetty-launcher-116-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 258731 INFO  (jetty-launcher-116-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 258745 INFO  (jetty-launcher-116-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 258745 INFO  (jetty-launcher-116-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 258745 INFO  (jetty-launcher-116-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 258746 INFO  (jetty-launcher-116-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1113ab73{/solr,null,AVAILABLE}
   [junit4]   2> 258747 WARN  (jetty-launcher-116-thread-2) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@65c05866[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 258781 INFO  (jetty-launcher-116-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@238850ac{/solr,null,AVAILABLE}
   [junit4]   2> 258781 WARN  (jetty-launcher-116-thread-1) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@379c2f5f[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 258801 INFO  (jetty-launcher-116-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@35b75195{SSL,[ssl, http/1.1]}{127.0.0.1:33177}
   [junit4]   2> 258801 INFO  (jetty-launcher-116-thread-2) [    ] o.e.j.s.Server Started @258978ms
   [junit4]   2> 258801 INFO  (jetty-launcher-116-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33177}
   [junit4]   2> 258802 ERROR (jetty-launcher-116-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 258802 INFO  (jetty-launcher-116-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 258802 INFO  (jetty-launcher-116-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.2
   [junit4]   2> 258802 INFO  (jetty-launcher-116-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 258802 INFO  (jetty-launcher-116-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 258802 INFO  (jetty-launcher-116-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-19T03:47:55.110Z
   [junit4]   2> 258852 INFO  (jetty-launcher-116-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@16b8228{SSL,[ssl, http/1.1]}{127.0.0.1:36752}
   [junit4]   2> 258852 INFO  (jetty-launcher-116-thread-1) [    ] o.e.j.s.Server Started @259029ms
   [junit4]   2> 258852 INFO  (jetty-launcher-116-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36752}
   [junit4]   2> 258862 INFO  (zkConnectionManagerCallback-118-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 258870 ERROR (jetty-launcher-116-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 258870 INFO  (jetty-launcher-116-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 258870 INFO  (jetty-launcher-116-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.2
   [junit4]   2> 258870 INFO  (jetty-launcher-116-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 258870 INFO  (jetty-launcher-116-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 258870 INFO  (jetty-launcher-116-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-19T03:47:55.178Z
   [junit4]   2> 258909 INFO  (jetty-launcher-116-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 258947 INFO  (zkConnectionManagerCallback-120-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 259033 INFO  (jetty-launcher-116-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 262906 INFO  (jetty-launcher-116-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 262962 INFO  (jetty-launcher-116-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38092/solr
   [junit4]   2> 263066 INFO  (zkConnectionManagerCallback-124-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 263326 INFO  (zkConnectionManagerCallback-126-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 264035 INFO  (jetty-launcher-116-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 264129 INFO  (jetty-launcher-116-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38092/solr
   [junit4]   2> 264164 INFO  (zkConnectionManagerCallback-132-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 264386 INFO  (jetty-launcher-116-thread-2) [n:127.0.0.1:33177_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33177_solr
   [junit4]   2> 264387 INFO  (jetty-launcher-116-thread-2) [n:127.0.0.1:33177_solr    ] o.a.s.c.Overseer Overseer (id=74716722194546694-127.0.0.1:33177_solr-n_0000000000) starting
   [junit4]   2> 264420 INFO  (zkConnectionManagerCallback-134-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 264655 INFO  (zkConnectionManagerCallback-141-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 264672 INFO  (jetty-launcher-116-thread-1) [n:127.0.0.1:36752_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36752_solr
   [junit4]   2> 264693 INFO  (zkCallback-125-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 264693 INFO  (zkCallback-133-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 264708 INFO  (jetty-launcher-116-thread-2) [n:127.0.0.1:33177_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38092/solr ready
   [junit4]   2> 264764 INFO  (zkCallback-140-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 264770 INFO  (OverseerStateUpdate-74716722194546694-127.0.0.1:33177_solr-n_0000000000) [n:127.0.0.1:33177_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:33177_solr
   [junit4]   2> 264801 INFO  (jetty-launcher-116-thread-2) [n:127.0.0.1:33177_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:33177_solr as DOWN
   [junit4]   2> 264804 INFO  (jetty-launcher-116-thread-2) [n:127.0.0.1:33177_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 264804 INFO  (jetty-launcher-116-thread-2) [n:127.0.0.1:33177_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33177_solr
   [junit4]   2> 264875 INFO  (zkCallback-140-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 264875 INFO  (zkCallback-133-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 264878 INFO  (OverseerStateUpdate-74716722194546694-127.0.0.1:33177_solr-n_0000000000) [n:127.0.0.1:33177_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 265223 INFO  (jetty-launcher-116-thread-2) [n:127.0.0.1:33177_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 265349 INFO  (zkConnectionManagerCallback-147-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 265530 INFO  (jetty-launcher-116-thread-1) [n:127.0.0.1:36752_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 265531 INFO  (jetty-launcher-116-thread-1) [n:127.0.0.1:36752_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38092/solr ready
   [junit4]   2> 265551 INFO  (jetty-launcher-116-thread-1) [n:127.0.0.1:36752_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 265586 INFO  (jetty-launcher-116-thread-2) [n:127.0.0.1:33177_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33177.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66d3e70f
   [junit4]   2> 265658 INFO  (jetty-launcher-116-thread-2) [n:127.0.0.1:33177_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33177.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66d3e70f
   [junit4]   2> 265658 INFO  (jetty-launcher-116-thread-2) [n:127.0.0.1:33177_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33177.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66d3e70f
   [junit4]   2> 265710 INFO  (jetty-launcher-116-thread-2) [n:127.0.0.1:33177_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MigrateRouteKeyTest_565588B2122F9CD2-001/tempDir-001/node2/.
   [junit4]   2> 265746 INFO  (jetty-launcher-116-thread-1) [n:127.0.0.1:36752_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36752.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66d3e70f
   [junit4]   2> 265929 INFO  (jetty-launcher-116-thread-1) [n:127.0.0.1:36752_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36752.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66d3e70f
   [junit4]   2> 265930 INFO  (jetty-launcher-116-thread-1) [n:127.0.0.1:36752_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36752.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66d3e70f
   [junit4]   2> 265931 INFO  (jetty-launcher-116-thread-1) [n:127.0.0.1:36752_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MigrateRouteKeyTest_565588B2122F9CD2-001/tempDir-001/node1/.
   [junit4]   2> 266426 INFO  (zkConnectionManagerCallback-153-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 266510 INFO  (SUITE-MigrateRouteKeyTest-seed#[565588B2122F9CD2]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 266514 INFO  (SUITE-MigrateRouteKeyTest-seed#[565588B2122F9CD2]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38092/solr ready
   [junit4]   2> 266881 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterprop with params val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 266895 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2} status=0 QTime=14
   [junit4]   2> 266897 INFO  (SUITE-MigrateRouteKeyTest-seed#[565588B2122F9CD2]-worker) [    ] o.a.s.c.MigrateRouteKeyTest Using legacyCloud=false for cluster
   [junit4]   2> 267153 INFO  (TEST-MigrateRouteKeyTest.testMissingSplitKey-seed#[565588B2122F9CD2]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testMissingSplitKey
   [junit4]   2> 267171 INFO  (qtp1916398309-345) [n:127.0.0.1:33177_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf&name=testMissingSplitKey-source&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 267354 INFO  (OverseerThreadFactory-104-thread-1-processing-n:127.0.0.1:33177_solr) [n:127.0.0.1:33177_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection testMissingSplitKey-source
   [junit4]   2> 267608 INFO  (OverseerStateUpdate-74716722194546694-127.0.0.1:33177_solr-n_0000000000) [n:127.0.0.1:33177_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testMissingSplitKey-source",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testMissingSplitKey-source_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:33177/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 267914 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr    x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=testMissingSplitKey-source_shard1_replica_n1&action=CREATE&numShards=1&collection=testMissingSplitKey-source&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 269266 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.2
   [junit4]   2> 269340 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.s.IndexSchema [testMissingSplitKey-source_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 269360 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 269360 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testMissingSplitKey-source_shard1_replica_n1' using configuration from collection testMissingSplitKey-source, trusted=true
   [junit4]   2> 269361 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33177.solr.core.testMissingSplitKey-source.shard1.replica_n1' (registry 'solr.core.testMissingSplitKey-source.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66d3e70f
   [junit4]   2> 269362 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.c.SolrCore [[testMissingSplitKey-source_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MigrateRouteKeyTest_565588B2122F9CD2-001/tempDir-001/node2/testMissingSplitKey-source_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MigrateRouteKeyTest_565588B2122F9CD2-001/tempDir-001/node2/./testMissingSplitKey-source_shard1_replica_n1/data/]
   [junit4]   2> 269794 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 269794 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 269831 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 269831 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 269873 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@76f52ec0[testMissingSplitKey-source_shard1_replica_n1] main]
   [junit4]   2> 269875 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 269875 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 269876 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 269876 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1633930640028073984
   [junit4]   2> 269902 INFO  (searcherExecutor-113-thread-1-processing-n:127.0.0.1:33177_solr x:testMissingSplitKey-source_shard1_replica_n1 c:testMissingSplitKey-source s:shard1 r:core_node2) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.c.SolrCore [testMissingSplitKey-source_shard1_replica_n1] Registered new searcher Searcher@76f52ec0[testMissingSplitKey-source_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 269934 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testMissingSplitKey-source/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 269934 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/testMissingSplitKey-source/leaders/shard1
   [junit4]   2> 269937 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 269937 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 269937 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33177/solr/testMissingSplitKey-source_shard1_replica_n1/
   [junit4]   2> 269954 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 269955 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:33177/solr/testMissingSplitKey-source_shard1_replica_n1/ has no replicas
   [junit4]   2> 269955 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 269955 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/testMissingSplitKey-source/leaders/shard1/leader after winning as /collections/testMissingSplitKey-source/leader_elect/shard1/election/74716722194546694-core_node2-n_0000000000
   [junit4]   2> 269957 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:33177/solr/testMissingSplitKey-source_shard1_replica_n1/ shard1
   [junit4]   2> 270116 INFO  (zkCallback-125-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMissingSplitKey-source/state.json] for collection [testMissingSplitKey-source] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 270120 INFO  (zkCallback-125-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMissingSplitKey-source/state.json] for collection [testMissingSplitKey-source] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 270134 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 270136 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source s:shard1 r:core_node2 x:testMissingSplitKey-source_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=testMissingSplitKey-source_shard1_replica_n1&action=CREATE&numShards=1&collection=testMissingSplitKey-source&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2222
   [junit4]   2> 270228 INFO  (qtp1916398309-345) [n:127.0.0.1:33177_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 270335 INFO  (zkCallback-125-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMissingSplitKey-source/state.json] for collection [testMissingSplitKey-source] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 270335 INFO  (zkCallback-125-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMissingSplitKey-source/state.json] for collection [testMissingSplitKey-source] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 270371 INFO  (zkCallback-125-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMissingSplitKey-source/state.json] for collection [testMissingSplitKey-source] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 270408 INFO  (qtp1916398309-345) [n:127.0.0.1:33177_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf&name=testMissingSplitKey-source&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=3253
   [junit4]   2> 270460 INFO  (qtp1916398309-346) [n:127.0.0.1:33177_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf&name=testMissingSplitKey-target&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 270500 INFO  (OverseerThreadFactory-104-thread-2-processing-n:127.0.0.1:33177_solr) [n:127.0.0.1:33177_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection testMissingSplitKey-target
   [junit4]   2> 270517 INFO  (OverseerCollectionConfigSetProcessor-74716722194546694-127.0.0.1:33177_solr-n_0000000000) [n:127.0.0.1:33177_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 270808 INFO  (OverseerStateUpdate-74716722194546694-127.0.0.1:33177_solr-n_0000000000) [n:127.0.0.1:33177_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testMissingSplitKey-target",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testMissingSplitKey-target_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:33177/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 271106 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr    x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=testMissingSplitKey-target_shard1_replica_n1&action=CREATE&numShards=1&collection=testMissingSplitKey-target&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 272446 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.2
   [junit4]   2> 272554 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.s.IndexSchema [testMissingSplitKey-target_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 272557 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 272557 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testMissingSplitKey-target_shard1_replica_n1' using configuration from collection testMissingSplitKey-target, trusted=true
   [junit4]   2> 272558 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33177.solr.core.testMissingSplitKey-target.shard1.replica_n1' (registry 'solr.core.testMissingSplitKey-target.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66d3e70f
   [junit4]   2> 272571 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.c.SolrCore [[testMissingSplitKey-target_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MigrateRouteKeyTest_565588B2122F9CD2-001/tempDir-001/node2/testMissingSplitKey-target_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MigrateRouteKeyTest_565588B2122F9CD2-001/tempDir-001/node2/./testMissingSplitKey-target_shard1_replica_n1/data/]
   [junit4]   2> 273293 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 273293 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 273294 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 273294 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 273309 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@44de3bb9[testMissingSplitKey-target_shard1_replica_n1] main]
   [junit4]   2> 273322 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 273322 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 273323 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 273341 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1633930643661389824
   [junit4]   2> 273407 INFO  (searcherExecutor-118-thread-1-processing-n:127.0.0.1:33177_solr x:testMissingSplitKey-target_shard1_replica_n1 c:testMissingSplitKey-target s:shard1 r:core_node2) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.c.SolrCore [testMissingSplitKey-target_shard1_replica_n1] Registered new searcher Searcher@44de3bb9[testMissingSplitKey-target_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 273441 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testMissingSplitKey-target/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 273442 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/testMissingSplitKey-target/leaders/shard1
   [junit4]   2> 273515 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 273515 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 273515 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33177/solr/testMissingSplitKey-target_shard1_replica_n1/
   [junit4]   2> 273515 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 273515 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:33177/solr/testMissingSplitKey-target_shard1_replica_n1/ has no replicas
   [junit4]   2> 273515 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 273516 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/testMissingSplitKey-target/leaders/shard1/leader after winning as /collections/testMissingSplitKey-target/leader_elect/shard1/election/74716722194546694-core_node2-n_0000000000
   [junit4]   2> 273553 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:33177/solr/testMissingSplitKey-target_shard1_replica_n1/ shard1
   [junit4]   2> 273587 INFO  (zkCallback-125-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMissingSplitKey-target/state.json] for collection [testMissingSplitKey-target] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 273601 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 273654 INFO  (qtp1916398309-347) [n:127.0.0.1:33177_solr c:testMissingSplitKey-target s:shard1 r:core_node2 x:testMissingSplitKey-target_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=testMissingSplitKey-target_shard1_replica_n1&action=CREATE&numShards=1&collection=testMissingSplitKey-target&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2549
   [junit4]   2> 273674 INFO  (qtp1916398309-346) [n:127.0.0.1:33177_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 273675 INFO  (OverseerCollectionConfigSetProcessor-74716722194546694-127.0.0.1:33177_solr-n_0000000000) [n:127.0.0.1:33177_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 273762 INFO  (zkCallback-125-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMissingSplitKey-target/state.json] for collection [testMissingSplitKey-target] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 273762 INFO  (zkCallback-125-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMissingSplitKey-target/state.json] for collection [testMissingSplitKey-target] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 273763 INFO  (qtp1916398309-346) [n:127.0.0.1:33177_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf&name=testMissingSplitKey-target&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=3303
   [junit4]   2> 273784 INFO  (qtp1916398309-348) [n:127.0.0.1:33177_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :migrate with params split.key=&target.collection=testMissingSplitKey-target&forward.timeout=45&action=MIGRATE&collection=testMissingSplitKey-source&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 273952 ERROR (OverseerThreadFactory-104-thread-3-processing-n:127.0.0.1:33177_solr) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source   ] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: testMissingSplitKey-source operation: migrate failed:org.apache.solr.common.SolrException: The split.key cannot be null or empty
   [junit4]   2> at org.apache.solr.cloud.api.collections.MigrateCmd.call(MigrateCmd.java:97)
   [junit4]   2> at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:294)
   [junit4]   2> at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:505)
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 273974 INFO  (qtp1916398309-348) [n:127.0.0.1:33177_solr c:testMissingSplitKey-source   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={split.key=&target.collection=testMissingSplitKey-target&forward.timeout=45&action=MIGRATE&collection=testMissingSplitKey-source&wt=javabin&version=2} status=400 QTime=190
   [junit4]   2> 273989 INFO  (TEST-MigrateRouteKeyTest.testMissingSplitKey-seed#[565588B2122F9CD2]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testMissingSplitKey
   [junit4]   2> 274081 INFO  (TEST-MigrateRouteKeyTest.multipleShardMigrateTest-seed#[565588B2122F9CD2]) [    ] o.a.s.SolrTestCaseJ4 ###Starting multipleShardMigrateTest
   [junit4]   2> 274090 INFO  (qtp1916398309-349) [n:127.0.0.1:33177_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf&name=sourceCollection&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 274130 INFO  (OverseerCollectionConfigSetProcessor-74716722194546694-127.0.0.1:33177_solr-n_0000000000) [n:127.0.0.1:33177_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 274131 INFO  (OverseerThreadFactory-104-thread-4-processing-n:127.0.0.1:33177_solr) [n:127.0.0.1:33177_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection sourceCollection
   [junit4]   2> 274392 INFO  (OverseerStateUpdate-74716722194546694-127.0.0.1:33177_solr-n_0000000000) [n:127.0.0.1:33177_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"sourceCollection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"sourceCollection_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:33177/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 274408 INFO  (OverseerStateUpdate-74716722194546694-127.0.0.1:33177_solr-n_0000000000) [n:127.0.0.1:33177_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"sourceCollection",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"sourceCollection_shard2_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:36752/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 274615 INFO  (qtp1916398309-345) [n:127.0.0.1:33177_solr    x:sourceCollection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=sourceCollection_shard1_replica_n1&action=CREATE&numShards=2&collection=sourceCollection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 274863 INFO  (qtp401470118-339) [n:127.0.0.1:36752_solr    x:sourceCollection_shard2_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=sourceCollection_shard2_replica_n2&action=CREATE&numShards=2&collection=sourceCollection&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 274864 INFO  (qtp401470118-339) [n:127.0.0.1:36752_solr    x:sourceCollection_shard2_replica_n2] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 275956 INFO  (qtp1916398309-345) [n:127.0.0.1:33177_solr c:sourceCollection s:shard1 r:core_node3 x:sourceCollection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.2
   [junit4]   2> 275992 INFO  (qtp401470118-339) [n:127.0.0.1:36752_solr c:sourceCollection s:shard2 r:core_node4 x:sourceCollection_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.2
   [junit4]   2> 276045 INFO  (qtp401470118-339) [n:127.0.0.1:36752_solr c:sourceCollection s:shard2 r:core_node4 x:sourceCollection_shard2_replica_n2] o.a.s.s.IndexSchema [sourceCollection_shard2_replica_n2] Schema name=minimal
   [junit4]   2> 276065 INFO  (qtp1916398309-345) [n:127.0.0.1:33177_solr c:sourceCollection s:shard1 r:core_node3 x:sourceCollection_shard1_replica_n1] o.a.s.s.IndexSchema [sourceCollection_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 276081 INFO  (qtp401470118-339) [n:127.0.0.1:36752_solr c:sourceCollection s:shard2 r:core_node4 x:sourceCollection_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 276081 INFO  (qtp401470118-339) [n:127.0.0.1:36752_solr c:sourceCollection s:shard2 r:core_node4 x:sourceCollection_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'sourceCollection_shard2_replica_n2' using configuration from collection sourceCollection, trusted=true
   [junit4]   2> 276082 INFO  (qtp401470118-339) [n:127.0.0.1:36752_solr c:sourceCollection s:shard2 r:core_node4 x:sourceCollection_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36752.solr.core.sourceCollection.shard2.replica_n2' (registry 'solr.core.sourceCollection.shard2.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66d3e70f
   [junit4]   2> 276082 INFO  (qtp401470118-339) [n:127.0.0.1:36752_solr c:sourceCollection s:shard2 r:core_node4 x:sourceCollection_shard2_replica_n2] o.a.s.c.SolrCore [[sourceCollection_shard2_replica_n2] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MigrateRouteKeyTest_565588B2122F9CD2-001/tempDir-001/node1/sourceCollection_shard2_replica_n2], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MigrateRouteKeyTest_565588B2122F9CD2-001/tempDir-001/node1/./sourceCollection_shard2_replica_n2/data/]
   [junit4]   2> 276101 INFO  (qtp1916398309-345) [n:127.0.0.1:33177_solr c:sourceCollection s:shard1 r:core_node3 x:sourceCollection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 276101 INFO  (qtp1916398309-345) [n:127.0.0.1:33177_solr c:sourceCollection s:shard1 r:core_node3 x:sourceCollection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'sourceCollection_shard1_replica_n1' using configuration from collection sourceCollection, trusted=true
   [junit4]   2> 276101 INFO  (qtp1916398309-345) [n:127.0.0.1:33177_solr c:sourceCollection s:shard1 r:core_node3 x:sourceCollection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33177.solr.core.sourceCollection.shard1.replica_n1' (registry 'solr.core.sourceCollection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66d3e70f
   [junit4]   2> 276101 INFO  (qtp1916398309-345) [n:127.0.0.1:33177_solr c:sourceCollection s:shard1 r:core_node3 x:sourceCollection_shard1_replica_n1] o.a.s.c.SolrCore [[sourceCollection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MigrateRouteKeyTest_565588B2122F9CD2-001/tempDir-001/node2/sourceCollection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MigrateRouteKeyTest_565588B2122F9CD2-001/tempDir-001/node2/./sourceCollection_shard1_replica_n1/data/]
   [junit4]   2> 276654 INFO  (qtp1916398309-345) [n:127.0.0.1:33177_solr c:sourceCollection s:shard1 r:core_node3 x:sourceCollection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 276654 INFO  (qtp1916398309-345) [n:127.0.0.1:33177_solr c:sourceCollection s:shard1 r:core_node3 x:sourceCollection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 276655 INFO  (qtp1916398309-345) [n:127.0.0.1:33177_solr c:sourceCollection s:shard1 r:core_node3 x:sourceCollection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 276655 INFO  (qtp1916398309-345) [n:127.0.0.1:33177_solr c:sourceCollection s:shard1 r:core_node3 x:sourceCollection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 276657 INFO  (qtp1916398309-345) [n:127.0.0.

[...truncated too long message...]

e] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1125054605
     [copy] Copying 242 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1125054605
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1125054605

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 127ms :: artifacts dl 15ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 430 minutes 54 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1072)
                at hudson.FilePath.act(FilePath.java:1061)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1835)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1074)
        at hudson.FilePath.act(FilePath.java:1061)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 16 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.7/16/

2 tests failed.
FAILED:  org.apache.solr.cloud.OverseerTest.testShardLeaderChange

Error Message:


Stack Trace:
org.apache.solr.common.AlreadyClosedException
        at __randomizedtesting.SeedInfo.seed([C65F55BA4F749C69:180CD24D55EC6998]:0)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:69)
        at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:544)
        at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:112)
        at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93)
        at org.apache.solr.cloud.OverseerElectionContext.<init>(ElectionContext.java:825)
        at org.apache.solr.cloud.OverseerTest.electNewOverseer(OverseerTest.java:1420)
        at org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:905)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv

Error Message:
Some docs had errors -- check logs expected:<0> but was:<3>

Stack Trace:
java.lang.AssertionError: Some docs had errors -- check logs expected:<0> but was:<3>
        at __randomizedtesting.SeedInfo.seed([C65F55BA4F749C69:F04B37FCC529A678]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:834)
        at org.junit.Assert.assertEquals(Assert.java:645)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:352)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv(TestStressCloudBlindAtomicUpdates.java:207)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14936 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> 4667126 INFO  (SUITE-OverseerTest-seed#[C65F55BA4F749C69]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.OverseerTest_C65F55BA4F749C69-001/init-core-data-001
   [junit4]   2> 4667126 WARN  (SUITE-OverseerTest-seed#[C65F55BA4F749C69]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=3 numCloses=3
   [junit4]   2> 4667126 INFO  (SUITE-OverseerTest-seed#[C65F55BA4F749C69]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 4667144 INFO  (SUITE-OverseerTest-seed#[C65F55BA4F749C69]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 4667145 INFO  (SUITE-OverseerTest-seed#[C65F55BA4F749C69]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 4667145 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 4667145 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 4667253 INFO  (SUITE-OverseerTest-seed#[C65F55BA4F749C69]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:37637
   [junit4]   2> 4667253 INFO  (SUITE-OverseerTest-seed#[C65F55BA4F749C69]-worker) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:37637
   [junit4]   2> 4667253 INFO  (SUITE-OverseerTest-seed#[C65F55BA4F749C69]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 37637
   [junit4]   2> 4667272 INFO  (zkConnectionManagerCallback-7657-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4667318 INFO  (zkConnectionManagerCallback-7659-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4667318 INFO  (SUITE-OverseerTest-seed#[C65F55BA4F749C69]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 4667318 INFO  (SUITE-OverseerTest-seed#[C65F55BA4F749C69]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 4667326 INFO  (TEST-OverseerTest.testReplay-seed#[C65F55BA4F749C69]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testReplay
   [junit4]   2> 4667539 INFO  (zkConnectionManagerCallback-7663-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4668088 INFO  (zkConnectionManagerCallback-7669-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4668700 INFO  (zkConnectionManagerCallback-7674-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4668701 INFO  (TEST-OverseerTest.testReplay-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37637/solr ready
   [junit4]   2> 4668703 INFO  (TEST-OverseerTest.testReplay-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37637_solr
   [junit4]   2> 4668720 INFO  (TEST-OverseerTest.testReplay-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795842-127.0.0.1:37637_solr-n_0000000000) starting
   [junit4]   2> 4668739 INFO  (OverseerStateUpdate-74723108933795842-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37637_solr
   [junit4]   2> 4668767 DEBUG (OverseerAutoScalingTriggerThread-74723108933795842-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: []
   [junit4]   2> 4668777 DEBUG (ScheduledTrigger-7839-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 0 and last live nodes: 0
   [junit4]   2> 4669000 INFO  (closeThreadPool-7676-thread-1) [    ] o.a.s.c.Overseer Overseer (id=74723108933795842-127.0.0.1:37637_solr-n_0000000000) closing
   [junit4]   2> 4669000 INFO  (OverseerStateUpdate-74723108933795842-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37637_solr
   [junit4]   2> 4669001 INFO  (OverseerAutoScalingTriggerThread-74723108933795842-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 4669006 WARN  (OverseerExitThread) [    ] o.a.s.c.Overseer Error communicating with ZooKeeper
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:319) ~[java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4669097 INFO  (TEST-OverseerTest.testReplay-seed#[C65F55BA4F749C69]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testReplay
   [junit4]   2> 4669123 INFO  (TEST-OverseerTest.testStateChange-seed#[C65F55BA4F749C69]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateChange
   [junit4]   2> 4669270 INFO  (zkConnectionManagerCallback-7680-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4669273 INFO  (zkConnectionManagerCallback-7686-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4669276 INFO  (zkConnectionManagerCallback-7691-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4669277 INFO  (TEST-OverseerTest.testStateChange-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37637/solr ready
   [junit4]   2> 4669279 INFO  (TEST-OverseerTest.testStateChange-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37637_solr
   [junit4]   2> 4669280 INFO  (TEST-OverseerTest.testStateChange-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795845-127.0.0.1:37637_solr-n_0000000000) starting
   [junit4]   2> 4669293 INFO  (OverseerStateUpdate-74723108933795845-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37637_solr
   [junit4]   2> 4669345 DEBUG (OverseerAutoScalingTriggerThread-74723108933795845-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: []
   [junit4]   2> 4669346 DEBUG (ScheduledTrigger-7844-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 0 and last live nodes: 0
   [junit4]   2> 4669559 INFO  (closeThreadPool-7693-thread-1) [    ] o.a.s.c.Overseer Overseer (id=74723108933795845-127.0.0.1:37637_solr-n_0000000000) closing
   [junit4]   2> 4669559 INFO  (OverseerStateUpdate-74723108933795845-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer According to ZK I (id=74723108933795845-127.0.0.1:37637_solr-n_0000000000) am no longer a leader. propsId=null
   [junit4]   2> 4669559 INFO  (OverseerStateUpdate-74723108933795845-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37637_solr
   [junit4]   2> 4669560 INFO  (OverseerAutoScalingTriggerThread-74723108933795845-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 4669594 WARN  (OverseerExitThread) [    ] o.a.s.c.Overseer Error communicating with ZooKeeper
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:319) ~[java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4669736 INFO  (TEST-OverseerTest.testStateChange-seed#[C65F55BA4F749C69]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateChange
   [junit4]   2> 4669791 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDownNodeFailover
   [junit4]   2> 4669965 INFO  (zkConnectionManagerCallback-7695-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4669999 INFO  (zkConnectionManagerCallback-7701-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4670002 INFO  (zkConnectionManagerCallback-7706-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4670012 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37637/solr ready
   [junit4]   2> 4670016 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37637_solr
   [junit4]   2> 4670017 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795848-127.0.0.1:37637_solr-n_0000000000) starting
   [junit4]   2> 4670054 INFO  (OverseerStateUpdate-74723108933795848-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37637_solr
   [junit4]   2> 4670058 DEBUG (OverseerAutoScalingTriggerThread-74723108933795848-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: []
   [junit4]   2> 4670070 DEBUG (ScheduledTrigger-7849-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 0 and last live nodes: 0
   [junit4]   2> 4670080 INFO  (zkConnectionManagerCallback-7710-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4670100 INFO  (zkCallback-7694-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4670100 INFO  (zkCallback-7658-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4670101 INFO  (zkCallback-7705-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4670102 INFO  (zkCallback-7709-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4670102 INFO  (zkConnectionManagerCallback-7717-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4670103 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4670103 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37637/solr ready
   [junit4]   2> 4670240 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection0/leaders/shard1
   [junit4]   2> 4670254 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection0/leaders/shard1/leader after winning as /collections/collection0/leader_elect/shard1/election/74723108933795851-127.0.0.1_core1-n_0000000000
   [junit4]   2> 4670422 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard1
   [junit4]   2> 4670423 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/74723108933795851-127.0.0.1_core1-n_0000000000
   [junit4]   2> 4670576 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection2/leaders/shard1
   [junit4]   2> 4670577 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection2/leaders/shard1/leader after winning as /collections/collection2/leader_elect/shard1/election/74723108933795851-127.0.0.1_core1-n_0000000000
   [junit4]   2> 4670734 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection3/leaders/shard1
   [junit4]   2> 4670752 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection3/leaders/shard1/leader after winning as /collections/collection3/leader_elect/shard1/election/74723108933795851-127.0.0.1_core1-n_0000000000
   [junit4]   2> 4670917 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection4/leaders/shard1
   [junit4]   2> 4670934 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection4/leaders/shard1/leader after winning as /collections/collection4/leader_elect/shard1/election/74723108933795851-127.0.0.1_core1-n_0000000000
   [junit4]   2> 4671043 INFO  (zkConnectionManagerCallback-7719-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4671043 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795848-127.0.0.1:37637_solr-n_0000000000) closing
   [junit4]   2> 4671043 INFO  (OverseerAutoScalingTriggerThread-74723108933795848-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 4671051 INFO  (OverseerStateUpdate-74723108933795848-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer According to ZK I (id=74723108933795848-127.0.0.1:37637_solr-n_0000000000) am no longer a leader. propsId=null
   [junit4]   2> 4671051 INFO  (OverseerStateUpdate-74723108933795848-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37637_solr
   [junit4]   2> 4671067 WARN  (OverseerExitThread) [    ] o.a.s.c.Overseer Error communicating with ZooKeeper
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:319) ~[java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4671081 INFO  (zkConnectionManagerCallback-7725-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4671096 INFO  (zkConnectionManagerCallback-7730-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4671097 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4671098 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37637/solr ready
   [junit4]   2> 4671099 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37637_solr
   [junit4]   2> 4671099 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795853-127.0.0.1:37637_solr-n_0000000001) starting
   [junit4]   2> 4671124 INFO  (OverseerStateUpdate-74723108933795853-127.0.0.1:37637_solr-n_0000000001) [    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37637_solr
   [junit4]   2> 4671126 INFO  (OverseerStateUpdate-74723108933795853-127.0.0.1:37637_solr-n_0000000001) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4671153 INFO  (zkCallback-7716-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4671153 INFO  (zkCallback-7705-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4671153 INFO  (zkCallback-7729-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4671153 INFO  (zkCallback-7658-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4671153 INFO  (zkCallback-7718-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4671153 INFO  (zkCallback-7709-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4671155 DEBUG (OverseerAutoScalingTriggerThread-74723108933795853-127.0.0.1:37637_solr-n_0000000001) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: []
   [junit4]   2> 4671170 DEBUG (ScheduledTrigger-7854-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 0 and last live nodes: 0
   [junit4]   2> 4671201 INFO  (closeThreadPool-7732-thread-1) [    ] o.a.s.c.Overseer Overseer (id=74723108933795848-127.0.0.1:37637_solr-n_0000000000) closing
   [junit4]   2> 4671201 INFO  (closeThreadPool-7732-thread-2) [    ] o.a.s.c.Overseer Overseer (id=74723108933795853-127.0.0.1:37637_solr-n_0000000001) closing
   [junit4]   2> 4671201 INFO  (OverseerStateUpdate-74723108933795853-127.0.0.1:37637_solr-n_0000000001) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37637_solr
   [junit4]   2> 4671213 INFO  (OverseerAutoScalingTriggerThread-74723108933795853-127.0.0.1:37637_solr-n_0000000001) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 4671215 WARN  (OverseerExitThread) [    ] o.a.s.c.Overseer Error communicating with ZooKeeper
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:319) ~[java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4671344 INFO  (TEST-OverseerTest.testDownNodeFailover-seed#[C65F55BA4F749C69]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDownNodeFailover
   [junit4]   2> 4671362 INFO  (TEST-OverseerTest.testBadQueueItem-seed#[C65F55BA4F749C69]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testBadQueueItem
   [junit4]   2> 4671581 ERROR (Finalizer) [    ] o.a.s.c.CoreContainer CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=2013987747
   [junit4]   2> 4671581 ERROR (Finalizer) [    ] o.a.s.c.CoreContainer CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=1086989557
   [junit4]   2> 4671581 ERROR (Finalizer) [    ] o.a.s.c.CoreContainer CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=1236901821
   [junit4]   2> 4671582 ERROR (Finalizer) [    ] o.a.s.c.CoreContainer CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=949151302
   [junit4]   2> 4671629 INFO  (zkConnectionManagerCallback-7734-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4671645 INFO  (zkConnectionManagerCallback-7740-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4671659 INFO  (zkConnectionManagerCallback-7745-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4671661 INFO  (TEST-OverseerTest.testBadQueueItem-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37637/solr ready
   [junit4]   2> 4671662 INFO  (TEST-OverseerTest.testBadQueueItem-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37637_solr
   [junit4]   2> 4671671 INFO  (TEST-OverseerTest.testBadQueueItem-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795856-127.0.0.1:37637_solr-n_0000000000) starting
   [junit4]   2> 4671686 INFO  (OverseerStateUpdate-74723108933795856-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37637_solr
   [junit4]   2> 4671687 INFO  (zkConnectionManagerCallback-7749-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4671690 INFO  (zkCallback-7658-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4671690 INFO  (zkCallback-7748-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4671718 INFO  (zkCallback-7744-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4671718 INFO  (OverseerStateUpdate-74723108933795856-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4671745 DEBUG (OverseerAutoScalingTriggerThread-74723108933795856-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1]
   [junit4]   2> 4671747 DEBUG (ScheduledTrigger-7859-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1
   [junit4]   2> 4671860 INFO  (TEST-OverseerTest.testBadQueueItem-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard1
   [junit4]   2> 4671861 INFO  (TEST-OverseerTest.testBadQueueItem-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/74723108933795859-127.0.0.1_core1-n_0000000000
   [junit4]   2> 4672088 INFO  (TEST-OverseerTest.testBadQueueItem-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard2
   [junit4]   2> 4672107 INFO  (TEST-OverseerTest.testBadQueueItem-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard2/leader after winning as /collections/collection1/leader_elect/shard2/election/74723108933795859-127.0.0.1_core2-n_0000000000
   [junit4]   2> 4672241 INFO  (TEST-OverseerTest.testBadQueueItem-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard3
   [junit4]   2> 4672250 INFO  (TEST-OverseerTest.testBadQueueItem-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard3/leader after winning as /collections/collection1/leader_elect/shard3/election/74723108933795859-127.0.0.1_core3-n_0000000000
   [junit4]   2> 4672379 ERROR (OverseerStateUpdate-74723108933795856-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.o.CollectionMutator Skipping invalid Overseer message because it has no collection specified: {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "node_name":"127.0.0.1",
   [junit4]   2>   "core":"core0",
   [junit4]   2>   "core_node_name":"node0",
   [junit4]   2>   "collection":"",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 4672393 ERROR (OverseerStateUpdate-74723108933795856-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.o.CollectionMutator Skipping invalid Overseer message because it has no collection specified: {
   [junit4]   2>   "operation":"deletecore",
   [junit4]   2>   "node_name":"127.0.0.1",
   [junit4]   2>   "core":"core0",
   [junit4]   2>   "core_node_name":"node0",
   [junit4]   2>   "collection":""}
   [junit4]   2> 4672531 INFO  (TEST-OverseerTest.testBadQueueItem-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection2/leaders/shard1
   [junit4]   2> 4672532 INFO  (TEST-OverseerTest.testBadQueueItem-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection2/leaders/shard1/leader after winning as /collections/collection2/leader_elect/shard1/election/74723108933795859-127.0.0.1_core1-n_0000000000
   [junit4]   2> 4672673 INFO  (TEST-OverseerTest.testBadQueueItem-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection2/leaders/shard2
   [junit4]   2> 4672674 INFO  (TEST-OverseerTest.testBadQueueItem-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection2/leaders/shard2/leader after winning as /collections/collection2/leader_elect/shard2/election/74723108933795859-127.0.0.1_core2-n_0000000000
   [junit4]   2> 4672748 DEBUG (ScheduledTrigger-7859-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1
   [junit4]   2> 4672809 INFO  (TEST-OverseerTest.testBadQueueItem-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection2/leaders/shard3
   [junit4]   2> 4672826 INFO  (TEST-OverseerTest.testBadQueueItem-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection2/leaders/shard3/leader after winning as /collections/collection2/leader_elect/shard3/election/74723108933795859-127.0.0.1_core3-n_0000000000
   [junit4]   2> 4672959 INFO  (zkCallback-7658-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4672959 INFO  (zkCallback-7733-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4672959 INFO  (zkCallback-7744-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4672960 INFO  (zkCallback-7748-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4672980 INFO  (closeThreadPool-7753-thread-1) [    ] o.a.s.c.Overseer Overseer (id=74723108933795856-127.0.0.1:37637_solr-n_0000000000) closing
   [junit4]   2> 4672980 INFO  (OverseerStateUpdate-74723108933795856-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37637_solr
   [junit4]   2> 4672980 INFO  (OverseerAutoScalingTriggerThread-74723108933795856-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 4672982 WARN  (OverseerExitThread) [    ] o.a.s.c.Overseer Error communicating with ZooKeeper
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:319) ~[java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4673088 INFO  (TEST-OverseerTest.testBadQueueItem-seed#[C65F55BA4F749C69]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testBadQueueItem
   [junit4]   2> 4673116 INFO  (TEST-OverseerTest.testLatchWatcher-seed#[C65F55BA4F749C69]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testLatchWatcher
   [junit4]   2> 4673252 INFO  (TEST-OverseerTest.testLatchWatcher-seed#[C65F55BA4F749C69]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testLatchWatcher
   [junit4]   2> 4673257 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testOverseerStatsReset
   [junit4]   2> 4673487 INFO  (zkConnectionManagerCallback-7759-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4673506 INFO  (zkCallback-7658-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4673508 INFO  (zkCallback-7758-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4673524 INFO  (zkConnectionManagerCallback-7766-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4673525 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4673525 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37637/solr ready
   [junit4]   2> 4673580 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37637_solr
   [junit4]   2> 4673581 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795841-127.0.0.1:37637_solr-n_0000000000) starting
   [junit4]   2> 4673598 INFO  (OverseerStateUpdate-74723108933795841-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37637_solr
   [junit4]   2> 4673666 DEBUG (OverseerAutoScalingTriggerThread-74723108933795841-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [node1]
   [junit4]   2> 4673681 DEBUG (ScheduledTrigger-7864-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1
   [junit4]   2> 4673768 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard1
   [junit4]   2> 4673769 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/74723108933795860-node1_core1-n_0000000000
   [junit4]   2> 4673770 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795841-127.0.0.1:37637_solr-n_0000000000) closing
   [junit4]   2> 4673770 INFO  (OverseerAutoScalingTriggerThread-74723108933795841-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 4673771 INFO  (OverseerStateUpdate-74723108933795841-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37637_solr
   [junit4]   2> 4673772 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795841-127.0.0.1:37637_solr-n_0000000000) closing
   [junit4]   2> 4673772 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795841-127.0.0.1:37637_solr-n_0000000000) closing
   [junit4]   2> 4673772 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795841-127.0.0.1:37637_solr-n_0000000000) closing
   [junit4]   2> 4673773 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37637_solr
   [junit4]   2> 4673786 WARN  (OverseerExitThread) [    ] o.a.s.c.Overseer I (id=74723108933795841-127.0.0.1:37637_solr-n_0000000000) am exiting, but I'm still the leader
   [junit4]   2> 4673786 ERROR (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.LeaderElector node exists
   [junit4]   2> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /overseer_elect/leader
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:122) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$makePath$8(SolrZkClient.java:545) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:544) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:488) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:475) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:842) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:171) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:136) [java/:?]
   [junit4]   2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:308) [java/:?]
   [junit4]   2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:217) [java/:?]
   [junit4]   2> at org.apache.solr.cloud.OverseerTest.testOverseerStatsReset(OverseerTest.java:757) [test/:?]
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_191]
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_191]
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_191]
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191]
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [java/:?]
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [java/:?]
   [junit4]   2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [java/:?]
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [java/:?]
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [java/:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [java/:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [java/:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [java/:?]
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [java/:?]
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [java/:?]
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [java/:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4673788 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795841-127.0.0.1:37637_solr-n_0000000000) closing
   [junit4]   2> 4673788 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795841-127.0.0.1:37637_solr-n_0000000000) closing
   [junit4]   2> 4673788 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795841-127.0.0.1:37637_solr-n_0000000000) closing
   [junit4]   2> 4673788 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795841-127.0.0.1:37637_solr-n_0000000000) closing
   [junit4]   2> 4673788 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37637_solr
   [junit4]   2> 4673789 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795841-127.0.0.1:37637_solr-n_0000000002) starting
   [junit4]   2> 4673804 INFO  (OverseerStateUpdate-74723108933795841-127.0.0.1:37637_solr-n_0000000002) [    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37637_solr
   [junit4]   2> 4673815 INFO  (zkCallback-7765-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4673815 INFO  (zkCallback-7758-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4673816 INFO  (zkCallback-7658-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4673830 WARN  (ScheduledTrigger-7868-thread-1) [    ] o.a.s.c.a.TriggerBase Exception getting trigger state '/autoscaling/triggerState/.scheduled_maintenance'
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/triggerState/.scheduled_maintenance
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.hasData(ZkDistribStateManager.java:59) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.TriggerBase.restoreState(TriggerBase.java:266) [java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.run(ScheduledTriggers.java:621) [java/:?]
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4673831 WARN  (ScheduledTrigger-7868-thread-1) [    ] o.a.s.c.a.TriggerBase Exception updating trigger state '/autoscaling/triggerState/.scheduled_maintenance'
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/triggerState/.scheduled_maintenance
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.hasData(ZkDistribStateManager.java:59) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.TriggerBase.saveState(TriggerBase.java:246) [java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.run(ScheduledTriggers.java:643) [java/:?]
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4673850 INFO  (closeThreadPool-7770-thread-1) [    ] o.a.s.c.Overseer Overseer (id=74723108933795841-127.0.0.1:37637_solr-n_0000000002) closing
   [junit4]   2> 4673850 INFO  (OverseerStateUpdate-74723108933795841-127.0.0.1:37637_solr-n_0000000002) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37637_solr
   [junit4]   2> 4673858 WARN  (OverseerExitThread) [    ] o.a.s.c.Overseer I (id=74723108933795841-127.0.0.1:37637_solr-n_0000000002) am exiting, but I'm still the leader
   [junit4]   2> 4673887 WARN  (OverseerAutoScalingTriggerThread-74723108933795841-127.0.0.1:37637_solr-n_0000000002) [    ] o.a.s.c.a.OverseerTriggerThread Error removing old markers
   [junit4]   2> org.apache.solr.common.AlreadyClosedException: null
   [junit4]   2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:74) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:80) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.cloud.DistribStateManager.listTree(DistribStateManager.java:104) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.cloud.DistribStateManager.removeRecursively(DistribStateManager.java:123) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.removeMarkers(OverseerTriggerThread.java:264) [java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:256) [java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4673887 WARN  (OverseerAutoScalingTriggerThread-74723108933795841-127.0.0.1:37637_solr-n_0000000002) [    ] o.a.s.c.a.OverseerTriggerThread Error removing old markers
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/nodeAdded
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1541) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getChildren$4(SolrZkClient.java:346) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:346) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:69) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:80) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.cloud.DistribStateManager.listTree(DistribStateManager.java:104) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.cloud.DistribStateManager.removeRecursively(DistribStateManager.java:123) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.removeMarkers(OverseerTriggerThread.java:264) [java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:257) [java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4673887 WARN  (OverseerAutoScalingTriggerThread-74723108933795841-127.0.0.1:37637_solr-n_0000000002) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_191]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:188) [java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4673966 INFO  (TEST-OverseerTest.testOverseerStatsReset-seed#[C65F55BA4F749C69]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testOverseerStatsReset
   [junit4]   2> 4674003 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testOverseerFailure
   [junit4]   2> 4674237 INFO  (zkConnectionManagerCallback-7774-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4674268 INFO  (zkCallback-7658-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4674269 INFO  (zkCallback-7773-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4674269 INFO  (zkConnectionManagerCallback-7778-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4674274 INFO  (zkConnectionManagerCallback-7784-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4674290 INFO  (zkConnectionManagerCallback-7789-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4674291 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4674291 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37637/solr ready
   [junit4]   2> 4674318 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37637_solr
   [junit4]   2> 4674319 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795863-127.0.0.1:37637_solr-n_0000000000) starting
   [junit4]   2> 4674349 INFO  (OverseerStateUpdate-74723108933795863-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37637_solr
   [junit4]   2> 4674350 INFO  (OverseerStateUpdate-74723108933795863-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4674497 DEBUG (OverseerAutoScalingTriggerThread-74723108933795863-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [node1]
   [junit4]   2> 4674497 DEBUG (ScheduledTrigger-7873-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1
   [junit4]   2> 4674499 INFO  (zkConnectionManagerCallback-7794-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4674512 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4674513 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37637/solr ready
   [junit4]   2> 4674618 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard1
   [junit4]   2> 4674619 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/74723108933795862-node1_core1-n_0000000000
   [junit4]   2> 4674687 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard1
   [junit4]   2> 4674688 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/74723108933795862-node1_core1-n_0000000001
   [junit4]   2> 4674819 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard1
   [junit4]   2> 4674820 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/74723108933795862-node1_core1-n_0000000002
   [junit4]   2> 4674860 INFO  (zkConnectionManagerCallback-7796-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4674861 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795863-127.0.0.1:37637_solr-n_0000000000) closing
   [junit4]   2> 4674861 INFO  (OverseerStateUpdate-74723108933795863-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37637_solr
   [junit4]   2> 4674884 INFO  (OverseerAutoScalingTriggerThread-74723108933795863-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 4674885 WARN  (OverseerExitThread) [    ] o.a.s.c.Overseer Error communicating with ZooKeeper
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:319) ~[java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4674920 INFO  (zkConnectionManagerCallback-7802-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4674923 INFO  (zkConnectionManagerCallback-7807-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4674936 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4674937 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37637/solr ready
   [junit4]   2> 4674938 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37637_solr
   [junit4]   2> 4674938 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795867-127.0.0.1:37637_solr-n_0000000001) starting
   [junit4]   2> 4674974 INFO  (OverseerStateUpdate-74723108933795867-127.0.0.1:37637_solr-n_0000000001) [    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37637_solr
   [junit4]   2> 4674975 INFO  (OverseerStateUpdate-74723108933795867-127.0.0.1:37637_solr-n_0000000001) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4675042 DEBUG (OverseerAutoScalingTriggerThread-74723108933795867-127.0.0.1:37637_solr-n_0000000001) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [node1]
   [junit4]   2> 4675044 DEBUG (ScheduledTrigger-7878-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1
   [junit4]   2> 4675150 INFO  (zkCallback-7773-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4675150 INFO  (zkCallback-7788-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4675150 INFO  (zkCallback-7793-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4675150 INFO  (zkCallback-7806-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4675163 INFO  (zkCallback-7658-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4675163 INFO  (zkCallback-7795-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4675204 INFO  (closeThreadPool-7809-thread-1) [    ] o.a.s.c.Overseer Overseer (id=74723108933795863-127.0.0.1:37637_solr-n_0000000000) closing
   [junit4]   2> 4675204 INFO  (closeThreadPool-7809-thread-2) [    ] o.a.s.c.Overseer Overseer (id=74723108933795867-127.0.0.1:37637_solr-n_0000000001) closing
   [junit4]   2> 4675205 INFO  (OverseerStateUpdate-74723108933795867-127.0.0.1:37637_solr-n_0000000001) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37637_solr
   [junit4]   2> 4675205 INFO  (OverseerAutoScalingTriggerThread-74723108933795867-127.0.0.1:37637_solr-n_0000000001) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 4675224 WARN  (OverseerExitThread) [    ] o.a.s.c.Overseer Error communicating with ZooKeeper
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:319) ~[java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4675328 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[C65F55BA4F749C69]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testOverseerFailure
   [junit4]   2> 4675355 INFO  (TEST-OverseerTest.testShardAssignment-seed#[C65F55BA4F749C69]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testShardAssignment
   [junit4]   2> 4675511 INFO  (zkConnectionManagerCallback-7811-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4675514 INFO  (zkConnectionManagerCallback-7817-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4675517 INFO  (zkConnectionManagerCallback-7822-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4675518 INFO  (TEST-OverseerTest.testShardAssignment-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37637/solr ready
   [junit4]   2> 4675536 INFO  (TEST-OverseerTest.testShardAssignment-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37637_solr
   [junit4]   2> 4675537 INFO  (TEST-OverseerTest.testShardAssignment-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795870-127.0.0.1:37637_solr-n_0000000000) starting
   [junit4]   2> 4675542 INFO  (OverseerStateUpdate-74723108933795870-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37637_solr
   [junit4]   2> 4675554 INFO  (zkConnectionManagerCallback-7826-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4675555 DEBUG (OverseerAutoScalingTriggerThread-74723108933795870-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: []
   [junit4]   2> 4675568 DEBUG (ScheduledTrigger-7883-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 0 and last live nodes: 0
   [junit4]   2> 4675571 INFO  (zkCallback-7658-thread-4) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4675571 INFO  (zkCallback-7810-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4675571 INFO  (zkCallback-7821-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4675571 INFO  (zkCallback-7825-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4675685 INFO  (TEST-OverseerTest.testShardAssignment-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard1
   [junit4]   2> 4675686 INFO  (TEST-OverseerTest.testShardAssignment-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/74723108933795873-127.0.0.1_core1-n_0000000000
   [junit4]   2> 4675797 INFO  (TEST-OverseerTest.testShardAssignment-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard2
   [junit4]   2> 4675798 INFO  (TEST-OverseerTest.testShardAssignment-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard2/leader after winning as /collections/collection1/leader_elect/shard2/election/74723108933795873-127.0.0.1_core2-n_0000000000
   [junit4]   2> 4675916 INFO  (TEST-OverseerTest.testShardAssignment-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard3
   [junit4]   2> 4675926 INFO  (TEST-OverseerTest.testShardAssignment-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard3/leader after winning as /collections/collection1/leader_elect/shard3/election/74723108933795873-127.0.0.1_core3-n_0000000000
   [junit4]   2> 4676032 INFO  (TEST-OverseerTest.testShardAssignment-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard1
   [junit4]   2> 4676171 INFO  (TEST-OverseerTest.testShardAssignment-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard2
   [junit4]   2> 4676317 INFO  (TEST-OverseerTest.testShardAssignment-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard3
   [junit4]   2> 4676363 INFO  (TEST-OverseerTest.testShardAssignment-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration.
   [junit4]   2> 4676363 INFO  (TEST-OverseerTest.testShardAssignment-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration.
   [junit4]   2> 4676363 INFO  (TEST-OverseerTest.testShardAssignment-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration.
   [junit4]   2> 4676364 WARN  (zkCallback-7825-thread-1) [    ] o.a.s.c.LeaderElector Our node is no longer in line to be leader
   [junit4]   2> 4676364 WARN  (zkCallback-7825-thread-1) [    ] o.a.s.c.LeaderElector Our node is no longer in line to be leader
   [junit4]   2> 4676365 WARN  (zkCallback-7825-thread-1) [    ] o.a.s.c.LeaderElector Our node is no longer in line to be leader
   [junit4]   2> 4676366 INFO  (zkCallback-7658-thread-4) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4676366 INFO  (zkCallback-7810-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4676366 INFO  (zkCallback-7821-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4676383 INFO  (zkCallback-7825-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 4676441 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10978458c830020, likely client has closed socket
   [junit4]   2> 4676441 INFO  (closeThreadPool-7830-thread-1) [    ] o.a.s.c.Overseer Overseer (id=74723108933795870-127.0.0.1:37637_solr-n_0000000000) closing
   [junit4]   2> 4676441 INFO  (OverseerStateUpdate-74723108933795870-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37637_solr
   [junit4]   2> 4676442 INFO  (OverseerAutoScalingTriggerThread-74723108933795870-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 4676535 WARN  (OverseerExitThread) [    ] o.a.s.c.Overseer Error communicating with ZooKeeper
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:319) ~[java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4676763 INFO  (TEST-OverseerTest.testShardAssignment-seed#[C65F55BA4F749C69]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testShardAssignment
   [junit4]   2> 4676770 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[C65F55BA4F749C69]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testExternalClusterStateChangeBehavior
   [junit4]   2> 4676983 INFO  (zkConnectionManagerCallback-7834-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4676994 INFO  (zkConnectionManagerCallback-7840-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4676997 INFO  (zkConnectionManagerCallback-7845-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4677007 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37637/solr ready
   [junit4]   2> 4677008 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37637_solr
   [junit4]   2> 4677009 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[C65F55BA4F749C69]) [    ] o.a.s.c.Overseer Overseer (id=74723108933795874-127.0.0.1:37637_solr-n_0000000000) starting
   [junit4]   2> 4677031 INFO  (OverseerStateUpdate-74723108933795874-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37637_solr
   [junit4]   2> 4677036 DEBUG (OverseerAutoScalingTriggerThread-74723108933795874-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: []
   [junit4]   2> 4677036 DEBUG (ScheduledTrigger-7888-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 0 and last live nodes: 0
   [junit4]   2> 4677165 INFO  (OverseerStateUpdate-74723108933795874-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"addreplica",
   [junit4]   2>   "collection":"test",
   [junit4]   2>   "shard":"x",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "state":"down"}
   [junit4]   2> 4677266 INFO  (zkCallback-7658-thread-4) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [0])
   [junit4]   2> 4677266 ERROR (OverseerStateUpdate-74723108933795874-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer Exception in Overseer main queue loop
   [junit4]   2> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /clusterstate.json
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1336) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$setData$6(SolrZkClient.java:370) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:370) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.overseer.ZkStateWriter.writePendingUpdates(ZkStateWriter.java:220) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:247) [java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4677271 WARN  (OverseerStateUpdate-74723108933795874-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.o.ClusterStateMutator Collection test already exists. exit
   [junit4]   2> 4677271 WARN  (OverseerStateUpdate-74723108933795874-127.0.0.1:37637_solr-n_0000000000) [    ] o.a.s.c.Overseer Solr cannot talk to ZK, exiting Overseer work queue loop
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue/qn-0000000004
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.ZkDistributedQueue.removeFirst(ZkDistributedQueue.java:537) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.ZkDistributedQueue.poll(ZkDistributedQueue.java:211) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:185) [java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4677271 INFO  (OverseerStateUpdate-74723108933795874-127.0.0.1:37637_solr-n_0000000000) [  

[...truncated too long message...]

urce] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null214645166
     [copy] Copying 242 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null214645166
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null214645166

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 103ms :: artifacts dl 35ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 470 minutes 42 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1072)
                at hudson.FilePath.act(FilePath.java:1061)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1835)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1074)
        at hudson.FilePath.act(FilePath.java:1061)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 17 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.7/17/

3 tests failed.
FAILED:  org.apache.solr.cloud.LeaderTragicEventTest.test

Error Message:
Timeout waiting for new replica become leader Timeout waiting to see state for collection=collection1 :DocCollection(collection1//collections/collection1/state.json/5)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node3":{           "core":"collection1_shard1_replica_n1",           "base_url":"http://127.0.0.1:46233/solr",           "node_name":"127.0.0.1:46233_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false"},         "core_node4":{           "core":"collection1_shard1_replica_n2",           "base_url":"http://127.0.0.1:45213/solr",           "node_name":"127.0.0.1:45213_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0"} Live Nodes: [127.0.0.1:45213_solr, 127.0.0.1:46233_solr] Last available state: DocCollection(collection1//collections/collection1/state.json/5)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node3":{           "core":"collection1_shard1_replica_n1",           "base_url":"http://127.0.0.1:46233/solr",           "node_name":"127.0.0.1:46233_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false"},         "core_node4":{           "core":"collection1_shard1_replica_n2",           "base_url":"http://127.0.0.1:45213/solr",           "node_name":"127.0.0.1:45213_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Timeout waiting for new replica become leader
Timeout waiting to see state for collection=collection1 :DocCollection(collection1//collections/collection1/state.json/5)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "core":"collection1_shard1_replica_n1",
          "base_url":"http://127.0.0.1:46233/solr",
          "node_name":"127.0.0.1:46233_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node4":{
          "core":"collection1_shard1_replica_n2",
          "base_url":"http://127.0.0.1:45213/solr",
          "node_name":"127.0.0.1:45213_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
Live Nodes: [127.0.0.1:45213_solr, 127.0.0.1:46233_solr]
Last available state: DocCollection(collection1//collections/collection1/state.json/5)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "core":"collection1_shard1_replica_n1",
          "base_url":"http://127.0.0.1:46233/solr",
          "node_name":"127.0.0.1:46233_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node4":{
          "core":"collection1_shard1_replica_n2",
          "base_url":"http://127.0.0.1:45213/solr",
          "node_name":"127.0.0.1:45213_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([C25B89E5992E050B:4A0FB63F37D268F3]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:304)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:282)
        at org.apache.solr.cloud.LeaderTragicEventTest.test(LeaderTragicEventTest.java:84)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_stored_idx

Error Message:
expected:<456> but was:<461>

Stack Trace:
java.lang.AssertionError: expected:<456> but was:<461>
        at __randomizedtesting.SeedInfo.seed([C25B89E5992E050B:D23A241C8C8C4025]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:834)
        at org.junit.Assert.assertEquals(Assert.java:645)
        at org.junit.Assert.assertEquals(Assert.java:631)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:282)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_stored_idx(TestStressCloudBlindAtomicUpdates.java:251)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv

Error Message:
Some docs had errors -- check logs expected:<0> but was:<8>

Stack Trace:
java.lang.AssertionError: Some docs had errors -- check logs expected:<0> but was:<8>
        at __randomizedtesting.SeedInfo.seed([C25B89E5992E050B:F44FEBA313733F1A]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:834)
        at org.junit.Assert.assertEquals(Assert.java:645)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:352)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv(TestStressCloudBlindAtomicUpdates.java:207)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14710 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_C25B89E5992E050B-001/init-core-data-001
   [junit4]   2> 1856281 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[C25B89E5992E050B]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1856282 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[C25B89E5992E050B]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=SSL overhead seems to cause OutOfMemory when stress testing)
   [junit4]   2> 1856282 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[C25B89E5992E050B]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1856282 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[C25B89E5992E050B]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 10 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_C25B89E5992E050B-001/tempDir-001
   [junit4]   2> 1856283 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[C25B89E5992E050B]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1856283 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1856283 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1856383 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[C25B89E5992E050B]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:34998
   [junit4]   2> 1856383 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[C25B89E5992E050B]-worker) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:34998
   [junit4]   2> 1856383 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[C25B89E5992E050B]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 34998
   [junit4]   2> 1856389 INFO  (zkConnectionManagerCallback-7847-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1856394 INFO  (zkConnectionManagerCallback-7849-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1856396 INFO  (zkConnectionManagerCallback-7851-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1856398 WARN  (jetty-launcher-7852-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1856398 WARN  (jetty-launcher-7852-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1856399 INFO  (jetty-launcher-7852-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1856399 INFO  (jetty-launcher-7852-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1856399 INFO  (jetty-launcher-7852-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1856399 WARN  (jetty-launcher-7852-thread-3) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1856399 INFO  (jetty-launcher-7852-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1856399 INFO  (jetty-launcher-7852-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1856399 INFO  (jetty-launcher-7852-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1856399 INFO  (jetty-launcher-7852-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1856399 INFO  (jetty-launcher-7852-thread-3) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1856399 INFO  (jetty-launcher-7852-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1856400 WARN  (jetty-launcher-7852-thread-4) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1856400 INFO  (jetty-launcher-7852-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1856400 INFO  (jetty-launcher-7852-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1856400 INFO  (jetty-launcher-7852-thread-4) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1856401 INFO  (jetty-launcher-7852-thread-4) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1856401 INFO  (jetty-launcher-7852-thread-4) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1856401 INFO  (jetty-launcher-7852-thread-4) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1856402 INFO  (jetty-launcher-7852-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@74526c46{/solr,null,AVAILABLE}
   [junit4]   2> 1856403 WARN  (jetty-launcher-7852-thread-5) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1856403 WARN  (jetty-launcher-7852-thread-6) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1856403 INFO  (jetty-launcher-7852-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1856403 INFO  (jetty-launcher-7852-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1856403 INFO  (jetty-launcher-7852-thread-6) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1856403 INFO  (jetty-launcher-7852-thread-6) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1856403 INFO  (jetty-launcher-7852-thread-5) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1856403 INFO  (jetty-launcher-7852-thread-6) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1856404 WARN  (jetty-launcher-7852-thread-7) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1856404 WARN  (jetty-launcher-7852-thread-8) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1856404 INFO  (jetty-launcher-7852-thread-7) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1856404 INFO  (jetty-launcher-7852-thread-7) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1856404 INFO  (jetty-launcher-7852-thread-8) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1856404 INFO  (jetty-launcher-7852-thread-7) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1856404 INFO  (jetty-launcher-7852-thread-8) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1856404 INFO  (jetty-launcher-7852-thread-8) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1856404 WARN  (jetty-launcher-7852-thread-9) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1856405 INFO  (jetty-launcher-7852-thread-9) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1856405 INFO  (jetty-launcher-7852-thread-9) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1856405 INFO  (jetty-launcher-7852-thread-9) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1856405 WARN  (jetty-launcher-7852-thread-10) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1856406 INFO  (jetty-launcher-7852-thread-10) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1856406 INFO  (jetty-launcher-7852-thread-10) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1856406 INFO  (jetty-launcher-7852-thread-10) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1856406 INFO  (jetty-launcher-7852-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1856407 INFO  (jetty-launcher-7852-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1856407 INFO  (jetty-launcher-7852-thread-3) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1856407 INFO  (jetty-launcher-7852-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@20bdb7a1{/solr,null,AVAILABLE}
   [junit4]   2> 1856419 INFO  (jetty-launcher-7852-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@787a6753{HTTP/1.1,[http/1.1]}{127.0.0.1:33334}
   [junit4]   2> 1856419 INFO  (jetty-launcher-7852-thread-4) [    ] o.e.j.s.Server Started @1856483ms
   [junit4]   2> 1856419 INFO  (jetty-launcher-7852-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33334}
   [junit4]   2> 1856419 INFO  (jetty-launcher-7852-thread-5) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1856419 INFO  (jetty-launcher-7852-thread-5) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1856420 INFO  (jetty-launcher-7852-thread-5) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1856420 ERROR (jetty-launcher-7852-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1856420 INFO  (jetty-launcher-7852-thread-4) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1856420 INFO  (jetty-launcher-7852-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.2
   [junit4]   2> 1856420 INFO  (jetty-launcher-7852-thread-5) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@76644d0a{/solr,null,AVAILABLE}
   [junit4]   2> 1856420 INFO  (jetty-launcher-7852-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1856420 INFO  (jetty-launcher-7852-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1856420 INFO  (jetty-launcher-7852-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-21T22:54:09.985Z
   [junit4]   2> 1856421 INFO  (jetty-launcher-7852-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1856421 INFO  (jetty-launcher-7852-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1856421 INFO  (jetty-launcher-7852-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1856422 INFO  (jetty-launcher-7852-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6b06a44c{/solr,null,AVAILABLE}
   [junit4]   2> 1856423 INFO  (jetty-launcher-7852-thread-6) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1856423 INFO  (jetty-launcher-7852-thread-6) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1856423 INFO  (jetty-launcher-7852-thread-6) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1856424 INFO  (jetty-launcher-7852-thread-6) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7eddca83{/solr,null,AVAILABLE}
   [junit4]   2> 1856425 INFO  (jetty-launcher-7852-thread-7) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1856425 INFO  (jetty-launcher-7852-thread-7) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1856425 INFO  (jetty-launcher-7852-thread-7) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1856425 INFO  (jetty-launcher-7852-thread-7) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@59672f3d{/solr,null,AVAILABLE}
   [junit4]   2> 1856426 INFO  (jetty-launcher-7852-thread-8) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1856426 INFO  (jetty-launcher-7852-thread-8) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1856426 INFO  (jetty-launcher-7852-thread-8) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1856426 INFO  (jetty-launcher-7852-thread-8) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2895c25e{/solr,null,AVAILABLE}
   [junit4]   2> 1856428 INFO  (jetty-launcher-7852-thread-9) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1856428 INFO  (jetty-launcher-7852-thread-9) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1856428 INFO  (jetty-launcher-7852-thread-9) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1856428 INFO  (jetty-launcher-7852-thread-9) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7a153b0e{/solr,null,AVAILABLE}
   [junit4]   2> 1856429 INFO  (jetty-launcher-7852-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1856429 INFO  (jetty-launcher-7852-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1856429 INFO  (jetty-launcher-7852-thread-2) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1856429 INFO  (jetty-launcher-7852-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7baac980{/solr,null,AVAILABLE}
   [junit4]   2> 1856430 INFO  (jetty-launcher-7852-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@19c4fc8e{HTTP/1.1,[http/1.1]}{127.0.0.1:35695}
   [junit4]   2> 1856430 INFO  (jetty-launcher-7852-thread-1) [    ] o.e.j.s.Server Started @1856494ms
   [junit4]   2> 1856430 INFO  (jetty-launcher-7852-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35695}
   [junit4]   2> 1856431 ERROR (jetty-launcher-7852-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1856431 INFO  (jetty-launcher-7852-thread-10) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1856431 INFO  (jetty-launcher-7852-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1856431 INFO  (jetty-launcher-7852-thread-10) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1856431 INFO  (jetty-launcher-7852-thread-10) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1856431 INFO  (jetty-launcher-7852-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.2
   [junit4]   2> 1856431 INFO  (jetty-launcher-7852-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1856431 INFO  (jetty-launcher-7852-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1856431 INFO  (jetty-launcher-7852-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-21T22:54:09.996Z
   [junit4]   2> 1856431 INFO  (jetty-launcher-7852-thread-10) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7d6abcb1{/solr,null,AVAILABLE}
   [junit4]   2> 1856433 INFO  (jetty-launcher-7852-thread-6) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7660b4e8{HTTP/1.1,[http/1.1]}{127.0.0.1:36259}
   [junit4]   2> 1856433 INFO  (jetty-launcher-7852-thread-6) [    ] o.e.j.s.Server Started @1856497ms
   [junit4]   2> 1856433 INFO  (jetty-launcher-7852-thread-6) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36259}
   [junit4]   2> 1856433 INFO  (jetty-launcher-7852-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1be8fb31{HTTP/1.1,[http/1.1]}{127.0.0.1:46857}
   [junit4]   2> 1856433 INFO  (jetty-launcher-7852-thread-2) [    ] o.e.j.s.Server Started @1856497ms
   [junit4]   2> 1856433 ERROR (jetty-launcher-7852-thread-6) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1856433 INFO  (jetty-launcher-7852-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46857}
   [junit4]   2> 1856433 INFO  (jetty-launcher-7852-thread-6) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1856433 INFO  (jetty-launcher-7852-thread-10) [    ] o.e.j.s.AbstractConnector Started ServerConnector@22920d3c{HTTP/1.1,[http/1.1]}{127.0.0.1:40511}
   [junit4]   2> 1856433 INFO  (jetty-launcher-7852-thread-10) [    ] o.e.j.s.Server Started @1856497ms
   [junit4]   2> 1856433 INFO  (jetty-launcher-7852-thread-10) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40511}
   [junit4]   2> 1856433 ERROR (jetty-launcher-7852-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1856433 INFO  (jetty-launcher-7852-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1856433 ERROR (jetty-launcher-7852-thread-10) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1856433 INFO  (jetty-launcher-7852-thread-9) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7556df7{HTTP/1.1,[http/1.1]}{127.0.0.1:40574}
   [junit4]   2> 1856434 INFO  (jetty-launcher-7852-thread-9) [    ] o.e.j.s.Server Started @1856497ms
   [junit4]   2> 1856434 INFO  (jetty-launcher-7852-thread-9) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40574}
   [junit4]   2> 1856434 ERROR (jetty-launcher-7852-thread-9) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1856434 INFO  (jetty-launcher-7852-thread-9) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1856434 INFO  (jetty-launcher-7852-thread-8) [    ] o.e.j.s.AbstractConnector Started ServerConnector@50a75381{HTTP/1.1,[http/1.1]}{127.0.0.1:35108}
   [junit4]   2> 1856434 INFO  (jetty-launcher-7852-thread-8) [    ] o.e.j.s.Server Started @1856498ms
   [junit4]   2> 1856434 INFO  (jetty-launcher-7852-thread-8) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35108}
   [junit4]   2> 1856434 ERROR (jetty-launcher-7852-thread-8) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1856434 INFO  (jetty-launcher-7852-thread-8) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1856434 INFO  (jetty-launcher-7852-thread-7) [    ] o.e.j.s.AbstractConnector Started ServerConnector@60d2f4b3{HTTP/1.1,[http/1.1]}{127.0.0.1:44413}
   [junit4]   2> 1856434 INFO  (jetty-launcher-7852-thread-7) [    ] o.e.j.s.Server Started @1856498ms
   [junit4]   2> 1856435 INFO  (jetty-launcher-7852-thread-7) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44413}
   [junit4]   2> 1856435 ERROR (jetty-launcher-7852-thread-7) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1856435 INFO  (jetty-launcher-7852-thread-7) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1856433 INFO  (jetty-launcher-7852-thread-10) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1856434 INFO  (jetty-launcher-7852-thread-5) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2263bd69{HTTP/1.1,[http/1.1]}{127.0.0.1:43879}
   [junit4]   2> 1856435 INFO  (jetty-launcher-7852-thread-5) [    ] o.e.j.s.Server Started @1856499ms
   [junit4]   2> 1856435 INFO  (jetty-launcher-7852-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43879}
   [junit4]   2> 1856435 ERROR (jetty-launcher-7852-thread-5) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1856435 INFO  (jetty-launcher-7852-thread-5) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1856435 INFO  (jetty-launcher-7852-thread-5) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.2
   [junit4]   2> 1856435 INFO  (jetty-launcher-7852-thread-5) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1856435 INFO  (jetty-launcher-7852-thread-5) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1856436 INFO  (jetty-launcher-7852-thread-5) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-21T22:54:10.001Z
   [junit4]   2> 1856436 INFO  (jetty-launcher-7852-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7c27f57b{HTTP/1.1,[http/1.1]}{127.0.0.1:43358}
   [junit4]   2> 1856436 INFO  (jetty-launcher-7852-thread-3) [    ] o.e.j.s.Server Started @1856500ms
   [junit4]   2> 1856436 INFO  (jetty-launcher-7852-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43358}
   [junit4]   2> 1856436 ERROR (jetty-launcher-7852-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1856436 INFO  (jetty-launcher-7852-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1856436 INFO  (jetty-launcher-7852-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.2
   [junit4]   2> 1856436 INFO  (jetty-launcher-7852-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1856436 INFO  (jetty-launcher-7852-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1856436 INFO  (jetty-launcher-7852-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-21T22:54:10.001Z
   [junit4]   2> 1856436 INFO  (jetty-launcher-7852-thread-8) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.2
   [junit4]   2> 1856436 INFO  (jetty-launcher-7852-thread-8) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1856437 INFO  (jetty-launcher-7852-thread-8) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1856437 INFO  (jetty-launcher-7852-thread-8) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-21T22:54:10.002Z
   [junit4]   2> 1856437 INFO  (jetty-launcher-7852-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.2
   [junit4]   2> 1856437 INFO  (jetty-launcher-7852-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1856437 INFO  (jetty-launcher-7852-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1856437 INFO  (jetty-launcher-7852-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-21T22:54:10.002Z
   [junit4]   2> 1856437 INFO  (jetty-launcher-7852-thread-7) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.2
   [junit4]   2> 1856437 INFO  (jetty-launcher-7852-thread-7) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1856437 INFO  (jetty-launcher-7852-thread-7) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1856437 INFO  (jetty-launcher-7852-thread-7) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-21T22:54:10.002Z
   [junit4]   2> 1856437 INFO  (jetty-launcher-7852-thread-6) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.2
   [junit4]   2> 1856437 INFO  (jetty-launcher-7852-thread-6) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1856437 INFO  (jetty-launcher-7852-thread-6) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1856437 INFO  (jetty-launcher-7852-thread-6) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-21T22:54:10.002Z
   [junit4]   2> 1856438 INFO  (jetty-launcher-7852-thread-10) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.2
   [junit4]   2> 1856438 INFO  (jetty-launcher-7852-thread-10) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1856438 INFO  (jetty-launcher-7852-thread-10) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1856438 INFO  (jetty-launcher-7852-thread-10) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-21T22:54:10.003Z
   [junit4]   2> 1856438 INFO  (jetty-launcher-7852-thread-9) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.2
   [junit4]   2> 1856438 INFO  (jetty-launcher-7852-thread-9) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1856438 INFO  (jetty-launcher-7852-thread-9) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1856438 INFO  (jetty-launcher-7852-thread-9) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-21T22:54:10.003Z
   [junit4]   2> 1856451 INFO  (zkConnectionManagerCallback-7854-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1856452 INFO  (zkConnectionManagerCallback-7866-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1856452 INFO  (jetty-launcher-7852-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1856454 INFO  (zkConnectionManagerCallback-7858-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1856455 INFO  (zkConnectionManagerCallback-7856-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1856456 INFO  (zkConnectionManagerCallback-7870-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1856456 INFO  (zkConnectionManagerCallback-7868-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1856457 INFO  (zkConnectionManagerCallback-7860-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1856457 INFO  (zkConnectionManagerCallback-7864-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1856458 INFO  (zkConnectionManagerCallback-7862-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1856458 INFO  (zkConnectionManagerCallback-7872-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1856463 INFO  (jetty-launcher-7852-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1856463 INFO  (jetty-launcher-7852-thread-5) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1856464 INFO  (jetty-launcher-7852-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1856464 INFO  (jetty-launcher-7852-thread-10) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1856464 INFO  (jetty-launcher-7852-thread-7) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1856465 INFO  (jetty-launcher-7852-thread-6) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1856465 INFO  (jetty-launcher-7852-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1856465 INFO  (jetty-launcher-7852-thread-8) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1856465 INFO  (jetty-launcher-7852-thread-9) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1856865 INFO  (jetty-launcher-7852-thread-6) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 1856882 INFO  (jetty-launcher-7852-thread-4) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 1856893 INFO  (jetty-launcher-7852-thread-8) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 1856946 INFO  (jetty-launcher-7852-thread-8) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34998/solr
   [junit4]   2> 1856972 INFO  (jetty-launcher-7852-thread-6) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34998/solr
   [junit4]   2> 1857039 INFO  (jetty-launcher-7852-thread-10) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 1857054 INFO  (jetty-launcher-7852-thread-9) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 1857055 INFO  (zkConnectionManagerCallback-7880-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857070 INFO  (jetty-launcher-7852-thread-3) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 1857080 INFO  (jetty-launcher-7852-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 1857088 INFO  (jetty-launcher-7852-thread-10) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34998/solr
   [junit4]   2> 1857112 INFO  (jetty-launcher-7852-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34998/solr
   [junit4]   2> 1857115 INFO  (jetty-launcher-7852-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34998/solr
   [junit4]   2> 1857123 INFO  (zkConnectionManagerCallback-7876-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857127 INFO  (zkConnectionManagerCallback-7889-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857129 INFO  (zkConnectionManagerCallback-7885-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857130 INFO  (jetty-launcher-7852-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34998/solr
   [junit4]   2> 1857160 INFO  (zkConnectionManagerCallback-7896-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857188 INFO  (jetty-launcher-7852-thread-9) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34998/solr
   [junit4]   2> 1857200 INFO  (zkConnectionManagerCallback-7899-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857207 INFO  (zkConnectionManagerCallback-7892-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857252 INFO  (jetty-launcher-7852-thread-6) [n:127.0.0.1:36259_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36259_solr
   [junit4]   2> 1857253 INFO  (jetty-launcher-7852-thread-6) [n:127.0.0.1:36259_solr    ] o.a.s.c.Overseer Overseer (id=73343017089564689-127.0.0.1:36259_solr-n_0000000000) starting
   [junit4]   2> 1857259 INFO  (zkConnectionManagerCallback-7903-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857265 INFO  (zkConnectionManagerCallback-7908-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857272 INFO  (jetty-launcher-7852-thread-10) [n:127.0.0.1:40511_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40511_solr
   [junit4]   2> 1857295 INFO  (jetty-launcher-7852-thread-5) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 1857295 INFO  (zkConnectionManagerCallback-7906-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857296 INFO  (zkConnectionManagerCallback-7917-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857308 INFO  (zkCallback-7888-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1857309 INFO  (jetty-launcher-7852-thread-4) [n:127.0.0.1:33334_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1857313 INFO  (jetty-launcher-7852-thread-8) [n:127.0.0.1:35108_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:35108_solr as DOWN
   [junit4]   2> 1857315 INFO  (jetty-launcher-7852-thread-8) [n:127.0.0.1:35108_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1857315 INFO  (jetty-launcher-7852-thread-4) [n:127.0.0.1:33334_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:33334_solr as DOWN
   [junit4]   2> 1857315 INFO  (jetty-launcher-7852-thread-8) [n:127.0.0.1:35108_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35108_solr
   [junit4]   2> 1857340 INFO  (zkCallback-7888-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1857340 INFO  (zkConnectionManagerCallback-7915-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857356 INFO  (jetty-launcher-7852-thread-5) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34998/solr
   [junit4]   2> 1857356 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1049115e0610015, likely client has closed socket
   [junit4]   2> 1857356 INFO  (jetty-launcher-7852-thread-4) [n:127.0.0.1:33334_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1857356 INFO  (jetty-launcher-7852-thread-4) [n:127.0.0.1:33334_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33334_solr
   [junit4]   2> 1857357 INFO  (zkCallback-7907-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1857357 INFO  (jetty-launcher-7852-thread-6) [n:127.0.0.1:36259_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1857358 INFO  (zkCallback-7888-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1857360 INFO  (jetty-launcher-7852-thread-7) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 1857363 INFO  (zkCallback-7902-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1857364 INFO  (zkCallback-7907-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1857365 INFO  (jetty-launcher-7852-thread-6) [n:127.0.0.1:36259_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34998/solr ready
   [junit4]   2> 1857367 INFO  (zkConnectionManagerCallback-7919-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857368 INFO  (zkCallback-7916-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 1857373 INFO  (jetty-launcher-7852-thread-2) [n:127.0.0.1:46857_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1857378 INFO  (jetty-launcher-7852-thread-2) [n:127.0.0.1:46857_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:46857_solr as DOWN
   [junit4]   2> 1857378 INFO  (jetty-launcher-7852-thread-2) [n:127.0.0.1:46857_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1857378 INFO  (jetty-launcher-7852-thread-2) [n:127.0.0.1:46857_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46857_solr
   [junit4]   2> 1857380 INFO  (zkCallback-7888-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1857380 INFO  (zkCallback-7907-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1857381 INFO  (zkConnectionManagerCallback-7925-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857382 INFO  (zkConnectionManagerCallback-7939-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857384 INFO  (zkCallback-7902-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1857384 INFO  (zkCallback-7916-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1857386 INFO  (jetty-launcher-7852-thread-7) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34998/solr
   [junit4]   2> 1857395 INFO  (zkCallback-7914-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (4)
   [junit4]   2> 1857396 INFO  (zkCallback-7918-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1857406 INFO  (OverseerStateUpdate-73343017089564689-127.0.0.1:36259_solr-n_0000000000) [n:127.0.0.1:36259_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:36259_solr
   [junit4]   2> 1857406 INFO  (jetty-launcher-7852-thread-3) [n:127.0.0.1:43358_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1857406 INFO  (jetty-launcher-7852-thread-6) [n:127.0.0.1:36259_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:36259_solr as DOWN
   [junit4]   2> 1857409 INFO  (jetty-launcher-7852-thread-6) [n:127.0.0.1:36259_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1857409 INFO  (jetty-launcher-7852-thread-6) [n:127.0.0.1:36259_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36259_solr
   [junit4]   2> 1857412 INFO  (zkCallback-7907-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1857412 INFO  (zkCallback-7916-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1857412 INFO  (zkCallback-7902-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1857412 INFO  (zkCallback-7888-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1857416 INFO  (jetty-launcher-7852-thread-3) [n:127.0.0.1:43358_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:43358_solr as DOWN
   [junit4]   2> 1857417 INFO  (jetty-launcher-7852-thread-3) [n:127.0.0.1:43358_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1857417 INFO  (jetty-launcher-7852-thread-3) [n:127.0.0.1:43358_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43358_solr
   [junit4]   2> 1857418 INFO  (zkCallback-7902-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 1857419 INFO  (zkCallback-7916-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 1857419 INFO  (zkCallback-7907-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 1857420 INFO  (zkCallback-7888-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 1857467 INFO  (zkConnectionManagerCallback-7942-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857471 INFO  (zkConnectionManagerCallback-7930-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857473 INFO  (jetty-launcher-7852-thread-10) [n:127.0.0.1:40511_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (6)
   [junit4]   2> 1857474 INFO  (jetty-launcher-7852-thread-9) [n:127.0.0.1:40574_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (6)
   [junit4]   2> 1857474 INFO  (jetty-launcher-7852-thread-10) [n:127.0.0.1:40511_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34998/solr ready
   [junit4]   2> 1857479 INFO  (jetty-launcher-7852-thread-9) [n:127.0.0.1:40574_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:40574_solr as DOWN
   [junit4]   2> 1857480 INFO  (jetty-launcher-7852-thread-9) [n:127.0.0.1:40574_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1857480 INFO  (jetty-launcher-7852-thread-9) [n:127.0.0.1:40574_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40574_solr
   [junit4]   2> 1857481 INFO  (zkCallback-7902-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 1857482 INFO  (zkCallback-7916-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 1857482 INFO  (zkCallback-7907-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 1857489 INFO  (zkConnectionManagerCallback-7947-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857495 INFO  (zkConnectionManagerCallback-7954-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857497 INFO  (jetty-launcher-7852-thread-4) [n:127.0.0.1:33334_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
   [junit4]   2> 1857498 INFO  (jetty-launcher-7852-thread-4) [n:127.0.0.1:33334_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34998/solr ready
   [junit4]   2> 1857511 INFO  (zkConnectionManagerCallback-7935-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857512 INFO  (jetty-launcher-7852-thread-8) [n:127.0.0.1:35108_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
   [junit4]   2> 1857514 INFO  (jetty-launcher-7852-thread-8) [n:127.0.0.1:35108_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34998/solr ready
   [junit4]   2> 1857519 INFO  (jetty-launcher-7852-thread-6) [n:127.0.0.1:36259_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1857535 INFO  (jetty-launcher-7852-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 1857536 INFO  (zkCallback-7914-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (7)
   [junit4]   2> 1857537 INFO  (zkCallback-7918-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (7)
   [junit4]   2> 1857546 INFO  (zkCallback-7924-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (7)
   [junit4]   2> 1857557 INFO  (jetty-launcher-7852-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34998/solr
   [junit4]   2> 1857559 INFO  (jetty-launcher-7852-thread-8) [n:127.0.0.1:35108_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1857565 INFO  (jetty-launcher-7852-thread-6) [n:127.0.0.1:36259_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36259.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1857571 INFO  (jetty-launcher-7852-thread-4) [n:127.0.0.1:33334_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1857578 INFO  (jetty-launcher-7852-thread-10) [n:127.0.0.1:40511_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1857581 INFO  (zkConnectionManagerCallback-7970-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857587 INFO  (zkCallback-7941-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 1857588 INFO  (zkCallback-7888-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 1857588 INFO  (zkCallback-7929-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 1857588 INFO  (jetty-launcher-7852-thread-6) [n:127.0.0.1:36259_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36259.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1857589 INFO  (jetty-launcher-7852-thread-6) [n:127.0.0.1:36259_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36259.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1857590 INFO  (jetty-launcher-7852-thread-5) [n:127.0.0.1:43879_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
   [junit4]   2> 1857590 INFO  (jetty-launcher-7852-thread-6) [n:127.0.0.1:36259_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_C25B89E5992E050B-001/tempDir-001/node6/.
   [junit4]   2> 1857591 INFO  (zkConnectionManagerCallback-7965-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857593 INFO  (jetty-launcher-7852-thread-3) [n:127.0.0.1:43358_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
   [junit4]   2> 1857604 INFO  (zkConnectionManagerCallback-7981-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857633 INFO  (jetty-launcher-7852-thread-3) [n:127.0.0.1:43358_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34998/solr ready
   [junit4]   2> 1857619 INFO  (zkConnectionManagerCallback-7985-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857641 INFO  (zkConnectionManagerCallback-7976-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857643 INFO  (jetty-launcher-7852-thread-9) [n:127.0.0.1:40574_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
   [junit4]   2> 1857645 INFO  (jetty-launcher-7852-thread-9) [n:127.0.0.1:40574_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34998/solr ready
   [junit4]   2> 1857651 INFO  (jetty-launcher-7852-thread-5) [n:127.0.0.1:43879_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:43879_solr as DOWN
   [junit4]   2> 1857655 INFO  (zkConnectionManagerCallback-7959-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857655 INFO  (jetty-launcher-7852-thread-3) [n:127.0.0.1:43358_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1857655 INFO  (jetty-launcher-7852-thread-8) [n:127.0.0.1:35108_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35108.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1857660 INFO  (jetty-launcher-7852-thread-5) [n:127.0.0.1:43879_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1857660 INFO  (jetty-launcher-7852-thread-5) [n:127.0.0.1:43879_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43879_solr
   [junit4]   2> 1857661 INFO  (jetty-launcher-7852-thread-2) [n:127.0.0.1:46857_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
   [junit4]   2> 1857662 INFO  (zkCallback-7902-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 1857662 INFO  (zkCallback-7916-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 1857662 INFO  (zkCallback-7914-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 1857663 INFO  (zkCallback-7888-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 1857663 INFO  (zkCallback-7918-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 1857663 INFO  (zkCallback-7907-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 1857664 INFO  (jetty-launcher-7852-thread-2) [n:127.0.0.1:46857_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34998/solr ready
   [junit4]   2> 1857665 INFO  (jetty-launcher-7852-thread-9) [n:127.0.0.1:40574_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1857669 INFO  (jetty-launcher-7852-thread-7) [n:127.0.0.1:44413_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (8)
   [junit4]   2> 1857673 INFO  (jetty-launcher-7852-thread-7) [n:127.0.0.1:44413_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:44413_solr as DOWN
   [junit4]   2> 1857680 INFO  (jetty-launcher-7852-thread-7) [n:127.0.0.1:44413_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1857680 INFO  (jetty-launcher-7852-thread-7) [n:127.0.0.1:44413_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44413_solr
   [junit4]   2> 1857681 INFO  (zkCallback-7888-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 1857682 INFO  (zkCallback-7918-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 1857682 INFO  (zkCallback-7914-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 1857682 INFO  (zkCallback-7907-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 1857682 INFO  (zkCallback-7916-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 1857682 INFO  (zkCallback-7902-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 1857697 INFO  (jetty-launcher-7852-thread-2) [n:127.0.0.1:46857_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1857721 INFO  (zkCallback-7924-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
   [junit4]   2> 1857721 INFO  (zkCallback-7941-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
   [junit4]   2> 1857721 INFO  (zkCallback-7969-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
   [junit4]   2> 1857722 INFO  (zkCallback-7929-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
   [junit4]   2> 1857722 INFO  (zkCallback-7975-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
   [junit4]   2> 1857722 INFO  (zkCallback-7958-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
   [junit4]   2> 1857722 INFO  (zkCallback-7964-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
   [junit4]   2> 1857725 INFO  (zkCallback-7953-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
   [junit4]   2> 1857725 INFO  (zkCallback-7934-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
   [junit4]   2> 1857739 INFO  (jetty-launcher-7852-thread-10) [n:127.0.0.1:40511_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40511.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1857744 INFO  (zkConnectionManagerCallback-7992-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857747 INFO  (zkCallback-7984-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 1857747 INFO  (zkConnectionManagerCallback-7997-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857750 INFO  (jetty-launcher-7852-thread-5) [n:127.0.0.1:43879_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (9)
   [junit4]   2> 1857752 INFO  (jetty-launcher-7852-thread-5) [n:127.0.0.1:43879_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34998/solr ready
   [junit4]   2> 1857761 INFO  (jetty-launcher-7852-thread-1) [n:127.0.0.1:35695_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (9)
   [junit4]   2> 1857766 INFO  (jetty-launcher-7852-thread-1) [n:127.0.0.1:35695_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:35695_solr as DOWN
   [junit4]   2> 1857767 INFO  (jetty-launcher-7852-thread-5) [n:127.0.0.1:43879_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1857767 INFO  (jetty-launcher-7852-thread-1) [n:127.0.0.1:35695_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1857767 INFO  (jetty-launcher-7852-thread-1) [n:127.0.0.1:35695_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35695_solr
   [junit4]   2> 1857771 INFO  (zkCallback-7902-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857771 INFO  (zkCallback-7907-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857771 INFO  (zkCallback-7918-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857771 INFO  (zkCallback-7924-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857772 INFO  (zkCallback-7914-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857772 INFO  (zkCallback-7969-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857772 INFO  (zkCallback-7929-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857772 INFO  (zkCallback-7916-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857772 INFO  (zkCallback-7975-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857772 INFO  (zkCallback-7888-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857773 INFO  (zkCallback-7941-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857773 INFO  (zkCallback-7984-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857781 INFO  (jetty-launcher-7852-thread-4) [n:127.0.0.1:33334_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33334.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1857782 INFO  (zkCallback-7953-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857783 INFO  (zkCallback-7996-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857783 INFO  (zkCallback-7991-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857787 INFO  (zkCallback-7958-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857788 INFO  (zkCallback-7964-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857790 INFO  (zkCallback-7934-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1857806 INFO  (zkConnectionManagerCallback-8005-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857808 INFO  (jetty-launcher-7852-thread-1) [n:127.0.0.1:35695_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (10)
   [junit4]   2> 1857809 INFO  (jetty-launcher-7852-thread-1) [n:127.0.0.1:35695_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34998/solr ready
   [junit4]   2> 1857816 INFO  (jetty-launcher-7852-thread-1) [n:127.0.0.1:35695_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1857816 INFO  (jetty-launcher-7852-thread-8) [n:127.0.0.1:35108_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35108.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1857820 INFO  (jetty-launcher-7852-thread-8) [n:127.0.0.1:35108_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35108.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1857831 INFO  (zkConnectionManagerCallback-8011-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1857833 INFO  (jetty-launcher-7852-thread-7) [n:127.0.0.1:44413_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (10)
   [junit4]   2> 1857834 INFO  (jetty-launcher-7852-thread-7) [n:127.0.0.1:44413_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34998/solr ready
   [junit4]   2> 1857835 INFO  (jetty-launcher-7852-thread-7) [n:127.0.0.1:44413_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1857835 INFO  (jetty-launcher-7852-thread-8) [n:127.0.0.1:35108_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_C25B89E5992E050B-001/tempDir-001/node8/.
   [junit4]   2> 1857914 INFO  (jetty-launcher-7852-thread-9) [n:127.0.0.1:40574_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40574.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1857925 INFO  (jetty-launcher-7852-thread-3) [n:127.0.0.1:43358_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43358.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1857926 INFO  (jetty-launcher-7852-thread-10) [n:127.0.0.1:40511_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40511.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1857926 INFO  (jetty-launcher-7852-thread-10) [n:127.0.0.1:40511_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40511.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1857928 INFO  (jetty-launcher-7852-thread-10) [n:127.0.0.1:40511_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_C25B89E5992E050B-001/tempDir-001/node10/.
   [junit4]   2> 1857937 INFO  (jetty-launcher-7852-thread-4) [n:127.0.0.1:33334_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33334.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1857938 INFO  (jetty-launcher-7852-thread-4) [n:127.0.0.1:33334_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33334.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1857939 INFO  (jetty-launcher-7852-thread-4) [n:127.0.0.1:33334_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_C25B89E5992E050B-001/tempDir-001/node4/.
   [junit4]   2> 1857940 INFO  (jetty-launcher-7852-thread-2) [n:127.0.0.1:46857_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46857.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1857940 INFO  (jetty-launcher-7852-thread-1) [n:127.0.0.1:35695_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35695.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1857945 INFO  (jetty-launcher-7852-thread-5) [n:127.0.0.1:43879_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43879.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1858032 INFO  (jetty-launcher-7852-thread-9) [n:127.0.0.1:40574_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40574.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1858034 INFO  (jetty-launcher-7852-thread-9) [n:127.0.0.1:40574_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40574.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1858036 INFO  (jetty-launcher-7852-thread-9) [n:127.0.0.1:40574_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_C25B89E5992E050B-001/tempDir-001/node9/.
   [junit4]   2> 1858038 INFO  (jetty-launcher-7852-thread-7) [n:127.0.0.1:44413_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44413.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1858040 INFO  (jetty-launcher-7852-thread-3) [n:127.0.0.1:43358_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43358.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1858040 INFO  (jetty-launcher-7852-thread-3) [n:127.0.0.1:43358_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43358.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1858042 INFO  (jetty-launcher-7852-thread-3) [n:127.0.0.1:43358_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_C25B89E5992E050B-001/tempDir-001/node3/.
   [junit4]   2> 1858044 INFO  (jetty-launcher-7852-thread-2) [n:127.0.0.1:46857_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46857.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1858045 INFO  (jetty-launcher-7852-thread-2) [n:127.0.0.1:46857_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46857.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1858048 INFO  (jetty-launcher-7852-thread-1) [n:127.0.0.1:35695_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35695.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1858048 INFO  (jetty-launcher-7852-thread-1) [n:127.0.0.1:35695_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35695.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1858050 INFO  (jetty-launcher-7852-thread-2) [n:127.0.0.1:46857_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_C25B89E5992E050B-001/tempDir-001/node2/.
   [junit4]   2> 1858050 INFO  (jetty-launcher-7852-thread-1) [n:127.0.0.1:35695_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_C25B89E5992E050B-001/tempDir-001/node1/.
   [junit4]   2> 1858056 INFO  (jetty-launcher-7852-thread-5) [n:127.0.0.1:43879_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43879.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1858065 INFO  (jetty-launcher-7852-thread-5) [n:127.0.0.1:43879_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43879.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1858070 INFO  (jetty-launcher-7852-thread-5) [n:127.0.0.1:43879_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_C25B89E5992E050B-001/tempDir-001/node5/.
   [junit4]   2> 1858072 INFO  (jetty-launcher-7852-thread-7) [n:127.0.0.1:44413_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44413.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1858073 INFO  (jetty-launcher-7852-thread-7) [n:127.0.0.1:44413_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44413.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@446c9cc8
   [junit4]   2> 1858075 INFO  (jetty-launcher-7852-thread-7) [n:127.0.0.1:44413_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_C25B89E5992E050B-001/tempDir-001/node7/.
   [junit4]   2> 1859439 INFO  (zkConnectionManagerCallback-8017-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1859447 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[C25B89E5992E050B]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (10)
   [junit4]   2> 1859451 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[C25B89E5992E050B]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34998/solr ready
   [junit4]   2> 1859996 INFO  (qtp733654633-55111) [n:127.0.0.1:40574_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&name=test_col&nrtReplicas=2&action=CREATE&numShards=5&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1860001 INFO  (OverseerThreadFactory-6930-thread-1-processing-n:127.0.0.1:36259_solr) [n:127.0.0.1:36259_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection test_col
   [junit4]   2> 1860121 INFO  (OverseerStateUpdate-73343017089564689-127.0.0.1:36259_solr-n_0000000000) [n:127.0.0.1:36259_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"test_col",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"test_col_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:46857/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1860125 INFO  (OverseerStateUpdate-73343017089564689-127.0.0.1:36259_solr-n_0000000000) [n:127.0.0.1:36259_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"test_col",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"test_col_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:40574/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1860129 INFO  (OverseerStateUpdate-73343017089564689-127.0.0.1:36259_solr-n_0000000000) [n:127.0.0.1:36259_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"test_col",
   [junit4]   2>   "shard":"sha

[...truncated too long message...]



ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null650428741
     [copy] Copying 242 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null650428741
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null650428741

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 52ms :: artifacts dl 4ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 299 minutes 51 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1072)
                at hudson.FilePath.act(FilePath.java:1061)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1835)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1074)
        at hudson.FilePath.act(FilePath.java:1061)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 18 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.7/18/

1 tests failed.
FAILED:  org.apache.lucene.index.TestConcurrentMergeScheduler.testFlushExceptions

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([730AC7E1D88016C6:C760943BD16080F2]:0)
        at org.junit.Assert.fail(Assert.java:86)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.junit.Assert.assertTrue(Assert.java:52)
        at org.apache.lucene.index.TestConcurrentMergeScheduler.testFlushExceptions(TestConcurrentMergeScheduler.java:128)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 991 lines...]
   [junit4] Suite: org.apache.lucene.index.TestConcurrentMergeScheduler
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestConcurrentMergeScheduler -Dtests.method=testFlushExceptions -Dtests.seed=730AC7E1D88016C6 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/test-data/enwiki.random.lines.txt -Dtests.locale=hr -Dtests.timezone=Asia/Pyongyang -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 0.10s J0 | TestConcurrentMergeScheduler.testFlushExceptions <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > at __randomizedtesting.SeedInfo.seed([730AC7E1D88016C6:C760943BD16080F2]:0)
   [junit4]    > at org.apache.lucene.index.TestConcurrentMergeScheduler.testFlushExceptions(TestConcurrentMergeScheduler.java:128)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/core/test/J0/temp/lucene.index.TestConcurrentMergeScheduler_730AC7E1D88016C6-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {field=PostingsFormat(name=Memory), id=PostingsFormat(name=LuceneFixedGap), content=Lucene50(blocksize=128)}, docValues:{}, maxPointsInLeafNode=1671, maxMBSortInHeap=6.217899396932761, sim=RandomSimilarity(queryNorm=false): {content=IB LL-DZ(0.3)}, locale=hr, timezone=Asia/Pyongyang
   [junit4]   2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=92748976,total=534773760
   [junit4]   2> NOTE: All tests run in this JVM: [TestExceedMaxTermLength, TestDocumentsWriterStallControl, TestDirectory, TestPersistentSnapshotDeletionPolicy, TestNumericDocValuesUpdates, Test4GBStoredFields, TestRegexpRandom, TestLongRangeFieldQueries, TestToken, TestMergedIterator, TestSimilarityProvider, TestMixedCodecs, TestTimSorterWorstCase, TestBinaryDocValuesUpdates, TestConcurrentMergeScheduler]
   [junit4] Completed [193/488 (1!)] on J0 in 22.42s, 16 tests, 1 failure <<< FAILURES!

[...truncated 64016 lines...]
[asciidoctor:convert] asciidoctor: ERROR: about-this-guide.adoc: line 1: invalid part, must have at least one section (e.g., chapter, appendix, etc.)
[asciidoctor:convert] asciidoctor: ERROR: solr-glossary.adoc: line 1: invalid part, must have at least one section (e.g., chapter, appendix, etc.)
     [java] Processed 2465 links (2015 relative) to 3226 anchors in 246 files
     [echo] Validated Links & Anchors via: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-ref-guide/bare-bones-html/

-documentation-lint:
    [jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/jtidy_tmp
     [echo] Checking for broken links...
     [exec]
     [exec] Crawl/parse...
     [exec]
     [exec] Verify...
     [echo] Checking for malformed docs...

jar-checksums:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

common.resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/null1636655966
     [copy] Copying 36 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/null1636655966
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/null1636655966

resolve-example:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

resolve-server:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1210438799
     [copy] Copying 242 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1210438799
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1210438799

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 53ms :: artifacts dl 8ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 261 minutes 19 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1072)
                at hudson.FilePath.act(FilePath.java:1061)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1835)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1074)
        at hudson.FilePath.act(FilePath.java:1061)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 18 - Still Unstable

Dawid Weiss-2
Hmm... this looks serious, no?
D.

On Thu, May 23, 2019 at 2:27 AM Apache Jenkins Server
<[hidden email]> wrote:

>
> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.7/18/
>
> 1 tests failed.
> FAILED:  org.apache.lucene.index.TestConcurrentMergeScheduler.testFlushExceptions
>
> Error Message:
>
>
> Stack Trace:
> java.lang.AssertionError
>         at __randomizedtesting.SeedInfo.seed([730AC7E1D88016C6:C760943BD16080F2]:0)
>         at org.junit.Assert.fail(Assert.java:86)
>         at org.junit.Assert.assertTrue(Assert.java:41)
>         at org.junit.Assert.assertTrue(Assert.java:52)
>         at org.apache.lucene.index.TestConcurrentMergeScheduler.testFlushExceptions(TestConcurrentMergeScheduler.java:128)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>         at java.lang.Thread.run(Thread.java:748)
>
>
>
>
> Build Log:
> [...truncated 991 lines...]
>    [junit4] Suite: org.apache.lucene.index.TestConcurrentMergeScheduler
>    [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestConcurrentMergeScheduler -Dtests.method=testFlushExceptions -Dtests.seed=730AC7E1D88016C6 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/test-data/enwiki.random.lines.txt -Dtests.locale=hr -Dtests.timezone=Asia/Pyongyang -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
>    [junit4] FAILURE 0.10s J0 | TestConcurrentMergeScheduler.testFlushExceptions <<<
>    [junit4]    > Throwable #1: java.lang.AssertionError
>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([730AC7E1D88016C6:C760943BD16080F2]:0)
>    [junit4]    >        at org.apache.lucene.index.TestConcurrentMergeScheduler.testFlushExceptions(TestConcurrentMergeScheduler.java:128)
>    [junit4]    >        at java.lang.Thread.run(Thread.java:748)
>    [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/core/test/J0/temp/lucene.index.TestConcurrentMergeScheduler_730AC7E1D88016C6-001
>    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {field=PostingsFormat(name=Memory), id=PostingsFormat(name=LuceneFixedGap), content=Lucene50(blocksize=128)}, docValues:{}, maxPointsInLeafNode=1671, maxMBSortInHeap=6.217899396932761, sim=RandomSimilarity(queryNorm=false): {content=IB LL-DZ(0.3)}, locale=hr, timezone=Asia/Pyongyang
>    [junit4]   2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=92748976,total=534773760
>    [junit4]   2> NOTE: All tests run in this JVM: [TestExceedMaxTermLength, TestDocumentsWriterStallControl, TestDirectory, TestPersistentSnapshotDeletionPolicy, TestNumericDocValuesUpdates, Test4GBStoredFields, TestRegexpRandom, TestLongRangeFieldQueries, TestToken, TestMergedIterator, TestSimilarityProvider, TestMixedCodecs, TestTimSorterWorstCase, TestBinaryDocValuesUpdates, TestConcurrentMergeScheduler]
>    [junit4] Completed [193/488 (1!)] on J0 in 22.42s, 16 tests, 1 failure <<< FAILURES!
>
> [...truncated 64016 lines...]
> [asciidoctor:convert] asciidoctor: ERROR: about-this-guide.adoc: line 1: invalid part, must have at least one section (e.g., chapter, appendix, etc.)
> [asciidoctor:convert] asciidoctor: ERROR: solr-glossary.adoc: line 1: invalid part, must have at least one section (e.g., chapter, appendix, etc.)
>      [java] Processed 2465 links (2015 relative) to 3226 anchors in 246 files
>      [echo] Validated Links & Anchors via: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-ref-guide/bare-bones-html/
>
> -documentation-lint:
>     [jtidy] Checking for broken html (such as invalid tags)...
>    [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/jtidy_tmp
>      [echo] Checking for broken links...
>      [exec]
>      [exec] Crawl/parse...
>      [exec]
>      [exec] Verify...
>      [echo] Checking for malformed docs...
>
> jar-checksums:
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> common.resolve:
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> jar-checksums:
>     [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/null1636655966
>      [copy] Copying 36 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/null1636655966
>    [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/null1636655966
>
> resolve-example:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> resolve-server:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> jar-checksums:
>     [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1210438799
>      [copy] Copying 242 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1210438799
>    [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1210438799
>
> check-working-copy:
> [ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
> [ivy:cachepath]         confs: [default]
> [ivy:cachepath]         found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
> [ivy:cachepath]         found com.jcraft#jsch;0.1.53 in public
> [ivy:cachepath]         found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
> [ivy:cachepath]         found org.apache.httpcomponents#httpclient;4.3.6 in public
> [ivy:cachepath]         found org.apache.httpcomponents#httpcore;4.3.3 in public
> [ivy:cachepath]         found commons-logging#commons-logging;1.1.3 in public
> [ivy:cachepath]         found commons-codec#commons-codec;1.6 in public
> [ivy:cachepath]         found org.slf4j#slf4j-api;1.7.2 in public
> [ivy:cachepath] :: resolution report :: resolve 53ms :: artifacts dl 8ms
>         ---------------------------------------------------------------------
>         |                  |            modules            ||   artifacts   |
>         |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
>         ---------------------------------------------------------------------
>         |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
>         ---------------------------------------------------------------------
> [wc-checker] Initializing working copy...
> [wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> [wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
> [wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
> [wc-checker] Checking working copy status...
>
> -jenkins-base:
>
> BUILD SUCCESSFUL
> Total time: 261 minutes 19 seconds
> Archiving artifacts
> java.lang.InterruptedException: no matches found within 10000
>         at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
>         at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
>         at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
>         at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
> Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
>                 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
>                 at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
>                 at hudson.remoting.Channel.call(Channel.java:955)
>                 at hudson.FilePath.act(FilePath.java:1072)
>                 at hudson.FilePath.act(FilePath.java:1061)
>                 at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
>                 at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
>                 at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
>                 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
>                 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
>                 at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
>                 at hudson.model.Build$BuildExecution.post2(Build.java:186)
>                 at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
>                 at hudson.model.Run.execute(Run.java:1835)
>                 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>                 at hudson.model.ResourceController.execute(ResourceController.java:97)
>                 at hudson.model.Executor.run(Executor.java:429)
> Caused: hudson.FilePath$TunneledInterruptedException
>         at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
>         at hudson.remoting.UserRequest.perform(UserRequest.java:212)
>         at hudson.remoting.UserRequest.perform(UserRequest.java:54)
>         at hudson.remoting.Request$2.run(Request.java:369)
>         at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:748)
> Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
>         at hudson.FilePath.act(FilePath.java:1074)
>         at hudson.FilePath.act(FilePath.java:1061)
>         at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
>         at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
>         at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
>         at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
>         at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
>         at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
>         at hudson.model.Build$BuildExecution.post2(Build.java:186)
>         at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
>         at hudson.model.Run.execute(Run.java:1835)
>         at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>         at hudson.model.ResourceController.execute(ResourceController.java:97)
>         at hudson.model.Executor.run(Executor.java:429)
> No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
> Recording test results
> Build step 'Publish JUnit test result report' changed build result to UNSTABLE
> Email was triggered for: Unstable (Test Failures)
> Sending email for trigger: Unstable (Test Failures)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 18 - Still Unstable

Jan Høydahl / Cominvent
I see this test failing now and then for the last year.
I have not studied the stack traces, can you say a few more words about what you found and why it may be serious?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

23. mai 2019 kl. 12:29 skrev Dawid Weiss <[hidden email]>:

Hmm... this looks serious, no?
D.

On Thu, May 23, 2019 at 2:27 AM Apache Jenkins Server
<[hidden email]> wrote:

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.7/18/

1 tests failed.
FAILED:  org.apache.lucene.index.TestConcurrentMergeScheduler.testFlushExceptions

Error Message:


Stack Trace:
java.lang.AssertionError
       at __randomizedtesting.SeedInfo.seed([730AC7E1D88016C6:C760943BD16080F2]:0)
       at org.junit.Assert.fail(Assert.java:86)
       at org.junit.Assert.assertTrue(Assert.java:41)
       at org.junit.Assert.assertTrue(Assert.java:52)
       at org.apache.lucene.index.TestConcurrentMergeScheduler.testFlushExceptions(TestConcurrentMergeScheduler.java:128)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
       at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
       at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
       at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
       at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
       at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
       at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
       at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
       at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
       at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
       at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
       at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
       at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
       at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
       at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
       at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
       at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
       at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
       at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
       at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
       at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
       at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
       at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
       at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
       at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
       at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
       at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
       at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
       at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
       at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
       at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
       at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 991 lines...]
  [junit4] Suite: org.apache.lucene.index.TestConcurrentMergeScheduler
  [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
  [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestConcurrentMergeScheduler -Dtests.method=testFlushExceptions -Dtests.seed=730AC7E1D88016C6 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/test-data/enwiki.random.lines.txt -Dtests.locale=hr -Dtests.timezone=Asia/Pyongyang -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
  [junit4] FAILURE 0.10s J0 | TestConcurrentMergeScheduler.testFlushExceptions <<<
  [junit4]    > Throwable #1: java.lang.AssertionError
  [junit4]    >        at __randomizedtesting.SeedInfo.seed([730AC7E1D88016C6:C760943BD16080F2]:0)
  [junit4]    >        at org.apache.lucene.index.TestConcurrentMergeScheduler.testFlushExceptions(TestConcurrentMergeScheduler.java:128)
  [junit4]    >        at java.lang.Thread.run(Thread.java:748)
  [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/core/test/J0/temp/lucene.index.TestConcurrentMergeScheduler_730AC7E1D88016C6-001
  [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {field=PostingsFormat(name=Memory), id=PostingsFormat(name=LuceneFixedGap), content=Lucene50(blocksize=128)}, docValues:{}, maxPointsInLeafNode=1671, maxMBSortInHeap=6.217899396932761, sim=RandomSimilarity(queryNorm=false): {content=IB LL-DZ(0.3)}, locale=hr, timezone=Asia/Pyongyang
  [junit4]   2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=92748976,total=534773760
  [junit4]   2> NOTE: All tests run in this JVM: [TestExceedMaxTermLength, TestDocumentsWriterStallControl, TestDirectory, TestPersistentSnapshotDeletionPolicy, TestNumericDocValuesUpdates, Test4GBStoredFields, TestRegexpRandom, TestLongRangeFieldQueries, TestToken, TestMergedIterator, TestSimilarityProvider, TestMixedCodecs, TestTimSorterWorstCase, TestBinaryDocValuesUpdates, TestConcurrentMergeScheduler]
  [junit4] Completed [193/488 (1!)] on J0 in 22.42s, 16 tests, 1 failure <<< FAILURES!

[...truncated 64016 lines...]
[asciidoctor:convert] asciidoctor: ERROR: about-this-guide.adoc: line 1: invalid part, must have at least one section (e.g., chapter, appendix, etc.)
[asciidoctor:convert] asciidoctor: ERROR: solr-glossary.adoc: line 1: invalid part, must have at least one section (e.g., chapter, appendix, etc.)
    [java] Processed 2465 links (2015 relative) to 3226 anchors in 246 files
    [echo] Validated Links & Anchors via: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-ref-guide/bare-bones-html/

-documentation-lint:
   [jtidy] Checking for broken html (such as invalid tags)...
  [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/jtidy_tmp
    [echo] Checking for broken links...
    [exec]
    [exec] Crawl/parse...
    [exec]
    [exec] Verify...
    [echo] Checking for malformed docs...

jar-checksums:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

common.resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
   [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/null1636655966
    [copy] Copying 36 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/null1636655966
  [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/null1636655966

resolve-example:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

resolve-server:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
   [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1210438799
    [copy] Copying 242 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1210438799
  [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1210438799

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath]         confs: [default]
[ivy:cachepath]         found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath]         found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath]         found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath]         found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath]         found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath]         found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath]         found commons-codec#commons-codec;1.6 in public
[ivy:cachepath]         found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 53ms :: artifacts dl 8ms
       ---------------------------------------------------------------------
       |                  |            modules            ||   artifacts   |
       |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
       ---------------------------------------------------------------------
       |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
       ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 261 minutes 19 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
       at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
       at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
       at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
       at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
               at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
               at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
               at hudson.remoting.Channel.call(Channel.java:955)
               at hudson.FilePath.act(FilePath.java:1072)
               at hudson.FilePath.act(FilePath.java:1061)
               at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
               at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
               at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
               at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
               at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
               at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
               at hudson.model.Build$BuildExecution.post2(Build.java:186)
               at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
               at hudson.model.Run.execute(Run.java:1835)
               at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
               at hudson.model.ResourceController.execute(ResourceController.java:97)
               at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
       at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
       at hudson.remoting.UserRequest.perform(UserRequest.java:212)
       at hudson.remoting.UserRequest.perform(UserRequest.java:54)
       at hudson.remoting.Request$2.run(Request.java:369)
       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
       at hudson.FilePath.act(FilePath.java:1074)
       at hudson.FilePath.act(FilePath.java:1061)
       at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
       at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
       at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
       at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
       at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
       at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
       at hudson.model.Build$BuildExecution.post2(Build.java:186)
       at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
       at hudson.model.Run.execute(Run.java:1835)
       at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
       at hudson.model.ResourceController.execute(ResourceController.java:97)
       at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)

---------------------------------------------------------------------
To unsubscribe, [hidden email]
For additional commands, [hidden email]

---------------------------------------------------------------------
To unsubscribe, [hidden email]
For additional commands, [hidden email]


Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 19 - Still Unstable

Apache Jenkins Server-2
In reply to this post by Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.7/19/

4 tests failed.
FAILED:  org.apache.lucene.classification.utils.ConfusionMatrixGeneratorTest.testGetConfusionMatrixWithBM25NB

Error Message:
expected:<7> but was:<6>

Stack Trace:
java.lang.AssertionError: expected:<7> but was:<6>
        at __randomizedtesting.SeedInfo.seed([4EAA3B5ECE16FEAD:F541BDBC1D7F1BE2]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:834)
        at org.junit.Assert.assertEquals(Assert.java:645)
        at org.junit.Assert.assertEquals(Assert.java:631)
        at org.apache.lucene.classification.utils.ConfusionMatrixGeneratorTest.checkCM(ConfusionMatrixGeneratorTest.java:110)
        at org.apache.lucene.classification.utils.ConfusionMatrixGeneratorTest.testGetConfusionMatrixWithBM25NB(ConfusionMatrixGeneratorTest.java:135)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.lucene.index.TestIndexWriterDelete.testDeletesOnDiskFull

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([46541D0A0C25A259]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterDelete

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([46541D0A0C25A259]:0)


FAILED:  org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:37038/y_xqp/bj

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:37038/y_xqp/bj
        at __randomizedtesting.SeedInfo.seed([71DC18D72A25C854:F988270D84D9A5AC]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.UnloadDistributedZkTest.testCoreUnloadAndLeaders(UnloadDistributedZkTest.java:308)
        at org.apache.solr.cloud.UnloadDistributedZkTest.test(UnloadDistributedZkTest.java:71)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1075)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1047)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 52 more




Build Log:
[...truncated 1685 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterDelete
   [junit4]   2> май 23, 2019 12:06:04 PM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.index.TestIndexWriterDelete
   [junit4]   2>    1) Thread[id=9, name=JUnit4-serializer-daemon, state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
   [junit4]   2>    2) Thread[id=1875, name=TEST-TestIndexWriterDelete.testDeletesOnDiskFull-seed#[46541D0A0C25A259], state=TIMED_WAITING, group=TGRP-TestIndexWriterDelete]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at org.apache.lucene.store.SlowClosingMockIndexInputWrapper.close(SlowClosingMockIndexInputWrapper.java:39)
   [junit4]   2>         at org.apache.lucene.util.IOUtils.close(IOUtils.java:88)
   [junit4]   2>         at org.apache.lucene.util.IOUtils.close(IOUtils.java:76)
   [junit4]   2>         at org.apache.lucene.codecs.lucene50.Lucene50PostingsReader.close(Lucene50PostingsReader.java:150)
   [junit4]   2>         at org.apache.lucene.util.IOUtils.close(IOUtils.java:88)
   [junit4]   2>         at org.apache.lucene.util.IOUtils.close(IOUtils.java:76)
   [junit4]   2>         at org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.close(BlockTreeTermsReader.java:249)
   [junit4]   2>         at org.apache.lucene.util.IOUtils.close(IOUtils.java:88)
   [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.close(PerFieldPostingsFormat.java:317)
   [junit4]   2>         at org.apache.lucene.util.IOUtils.close(IOUtils.java:88)
   [junit4]   2>         at org.apache.lucene.util.IOUtils.close(IOUtils.java:76)
   [junit4]   2>         at org.apache.lucene.index.SegmentCoreReaders.decRef(SegmentCoreReaders.java:170)
   [junit4]   2>         at org.apache.lucene.index.SegmentReader.doClose(SegmentReader.java:204)
   [junit4]   2>         at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:244)
   [junit4]   2>         at org.apache.lucene.index.IndexReader.close(IndexReader.java:382)
   [junit4]   2>         at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:787)
   [junit4]   2>         at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:300)
   [junit4]   2>         at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:285)
   [junit4]   2>         at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:281)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterDelete.doTestOperationsOnDiskFull(TestIndexWriterDelete.java:638)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterDelete.testDeletesOnDiskFull(TestIndexWriterDelete.java:486)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>    3) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1252)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1326)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:639)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:496)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:269)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:394)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2>    4) Thread[id=1874, name=SUITE-TestIndexWriterDelete-seed#[46541D0A0C25A259], state=RUNNABLE, group=TGRP-TestIndexWriterDelete]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1559)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:696)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:693)
   [junit4]   2>         at java.security.AccessController.doPrivileged(Native Method)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:708)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:629)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterDelete -Dtests.method=testDeletesOnDiskFull -Dtests.seed=46541D0A0C25A259 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/test-data/enwiki.random.lines.txt -Dtests.locale=be-BY -Dtests.timezone=Pacific/Samoa -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   7143s J2 | TestIndexWriterDelete.testDeletesOnDiskFull <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > at __randomizedtesting.SeedInfo.seed([46541D0A0C25A259]:0)
   [junit4]   2> май 23, 2019 12:06:04 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> май 23, 2019 12:06:24 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.lucene.index.TestIndexWriterDelete:
   [junit4]   2>    1) Thread[id=1875, name=TEST-TestIndexWriterDelete.testDeletesOnDiskFull-seed#[46541D0A0C25A259], state=RUNNABLE, group=TGRP-TestIndexWriterDelete]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at org.apache.lucene.store.SlowOpeningMockIndexInputWrapper.<init>(SlowOpeningMockIndexInputWrapper.java:33)
   [junit4]   2>         at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:769)
   [junit4]   2>         at org.apache.lucene.store.Directory.copyFrom(Directory.java:182)
   [junit4]   2>         at org.apache.lucene.util.TestUtil.ramCopyOf(TestUtil.java:1299)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterDelete.doTestOperationsOnDiskFull(TestIndexWriterDelete.java:528)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterDelete.testDeletesOnDiskFull(TestIndexWriterDelete.java:486)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> май 23, 2019 12:06:24 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=1875, name=TEST-TestIndexWriterDelete.testDeletesOnDiskFull-seed#[46541D0A0C25A259], state=TIMED_WAITING, group=TGRP-TestIndexWriterDelete]
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/core/test/J2/temp/lucene.index.TestIndexWriterDelete_46541D0A0C25A259-001
   [junit4]   2> May 23, 2019 11:06:25 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {field=Lucene50(blocksize=128), city=Lucene50(blocksize=128), contents=Lucene50(blocksize=128), id=Lucene50(blocksize=128), body=Lucene50(blocksize=128), value=PostingsFormat(name=Memory), content=PostingsFormat(name=Memory)}, docValues:{dv=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Memory), value=DocValuesFormat(name=Asserting), content=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=540, maxMBSortInHeap=5.061707069474123, sim=RandomSimilarity(queryNorm=false): {field=DFR G3(800.0), contents=DFR G2, city=DFR I(n)1, id=DFR GL3(800.0), body=IB LL-DZ(0.3), content=DFR GB3(800.0)}, locale=be-BY, timezone=Pacific/Samoa
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=175019680,total=237502464
   [junit4]   2> NOTE: All tests run in this JVM: [TestIndexWriterWithThreads, MultiCollectorTest, TestByteArrayDataInput, TestIndexOrDocValuesQuery, TestSpanCollection, TestLiveFieldValues, TestDirectMonotonic, TestMathUtil, TestFlex, TestOrdinalMap, TestBagOfPostings, TestLazyProxSkipping, TestMultiPhraseQuery, TestSimpleExplanationsOfNonMatches, TestSearcherManager, TestDocIdSetBuilder, TestScorerPerf, TestLRUQueryCache, TestSearchForDuplicates, TestPagedBytes, Test2BPoints, TestInfoStream, TestIndexWriterDelete]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterDelete -Dtests.seed=46541D0A0C25A259 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/test-data/enwiki.random.lines.txt -Dtests.locale=be-BY -Dtests.timezone=Pacific/Samoa -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | TestIndexWriterDelete (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > at __randomizedtesting.SeedInfo.seed([46541D0A0C25A259]:0)
   [junit4] Completed [340/488 (1!)] on J2 in 7220.75s, 15 tests, 2 errors <<< FAILURES!

[...truncated 6030 lines...]
   [junit4] Suite: org.apache.lucene.classification.utils.ConfusionMatrixGeneratorTest
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ConfusionMatrixGeneratorTest -Dtests.method=testGetConfusionMatrixWithBM25NB -Dtests.seed=4EAA3B5ECE16FEAD -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/test-data/enwiki.random.lines.txt -Dtests.locale=zh-HK -Dtests.timezone=Etc/UCT -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 8.93s J0 | ConfusionMatrixGeneratorTest.testGetConfusionMatrixWithBM25NB <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<7> but was:<6>
   [junit4]    > at __randomizedtesting.SeedInfo.seed([4EAA3B5ECE16FEAD:F541BDBC1D7F1BE2]:0)
   [junit4]    > at org.apache.lucene.classification.utils.ConfusionMatrixGeneratorTest.checkCM(ConfusionMatrixGeneratorTest.java:110)
   [junit4]    > at org.apache.lucene.classification.utils.ConfusionMatrixGeneratorTest.testGetConfusionMatrixWithBM25NB(ConfusionMatrixGeneratorTest.java:135)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/classification/test/J0/temp/lucene.classification.utils.ConfusionMatrixGeneratorTest_4EAA3B5ECE16FEAD-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {bool=PostingsFormat(name=Direct), cat=PostingsFormat(name=Direct), text=PostingsFormat(name=MockRandom)}, docValues:{}, maxPointsInLeafNode=1619, maxMBSortInHeap=6.587545159147714, sim=RandomSimilarity(queryNorm=false): {bool=IB SPL-L1, cat=DFR I(ne)3(800.0), text=DFR I(ne)L2}, locale=zh-HK, timezone=Etc/UCT
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=265193512,total=340262912
   [junit4]   2> NOTE: All tests run in this JVM: [DataSplitterTest, KNearestNeighborDocumentClassifierTest, KNearestNeighborClassifierTest, ConfusionMatrixGeneratorTest]
   [junit4] Completed [9/11 (1!)] on J0 in 14.67s, 7 tests, 1 failure <<< FAILURES!

[...truncated 6683 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest
   [junit4]   2> 2452776 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001/init-core-data-001
   [junit4]   2> 2452776 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 2452799 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2452821 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 2452821 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /y_xqp/bj
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 2453102 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 2453169 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2453206 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2453313 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_lucene2.us.west_apache_org_37141_hdfs____xpkid0/webapp
   [junit4]   2> 2454434 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.m.log Started HttpServer2$[hidden email]:37141
   [junit4]   2> 2455474 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2455475 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2455587 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_33293_datanode____.3hdcq5/webapp
   [junit4]   2> 2456702 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33293
   [junit4]   2> 2457523 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2457557 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2457720 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_46806_datanode____.98lllp/webapp
   [junit4]   2> 2458683 ERROR (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:45727) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 2458898 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x91acc12be56f65: from storage DS-94815bc6-fdbd-4292-aa0d-90dd9a1500d8 node DatanodeRegistration(127.0.0.1:36782, datanodeUuid=ea2b6405-7915-4786-855b-2d0a4542589c, infoPort=41995, infoSecurePort=0, ipcPort=32856, storageInfo=lv=-56;cid=testClusterID;nsid=1894079094;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 2458898 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x91acc12be56f65: from storage DS-011c5056-5fcf-474d-81c8-2c8ccba53ca2 node DatanodeRegistration(127.0.0.1:36782, datanodeUuid=ea2b6405-7915-4786-855b-2d0a4542589c, infoPort=41995, infoSecurePort=0, ipcPort=32856, storageInfo=lv=-56;cid=testClusterID;nsid=1894079094;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 2459365 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[71DC18D72A25C854]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46806
   [junit4]   2> 2460122 ERROR (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:45727) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 2460260 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x91acc17df854e8: from storage DS-3dbf43d4-deb6-4417-b83d-8b214fddfd22 node DatanodeRegistration(127.0.0.1:37081, datanodeUuid=4ee146d6-ac2b-439b-9727-2e6e10a345b2, infoPort=35405, infoSecurePort=0, ipcPort=45245, storageInfo=lv=-56;cid=testClusterID;nsid=1894079094;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 2460261 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x91acc17df854e8: from storage DS-e287b1d6-63c4-4f7e-9680-22af23ae36f8 node DatanodeRegistration(127.0.0.1:37081, datanodeUuid=4ee146d6-ac2b-439b-9727-2e6e10a345b2, infoPort=35405, infoSecurePort=0, ipcPort=45245, storageInfo=lv=-56;cid=testClusterID;nsid=1894079094;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 2460992 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2460992 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2460992 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2461092 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.ZkTestServer start zk server on port:42598
   [junit4]   2> 2461092 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:42598
   [junit4]   2> 2461092 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 42598
   [junit4]   2> 2461145 INFO  (zkConnectionManagerCallback-4345-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2461150 INFO  (zkConnectionManagerCallback-4347-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2461151 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2461166 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2461167 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2461168 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2461169 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2461178 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2461179 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2461180 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2461180 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2461181 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2461182 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2461183 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 2461689 WARN  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2461689 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2461689 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2461689 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2461693 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2461693 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2461693 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2461694 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@493b1f1f{/y_xqp/bj,null,AVAILABLE}
   [junit4]   2> 2461695 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@680cf7f4{HTTP/1.1,[http/1.1]}{127.0.0.1:37038}
   [junit4]   2> 2461695 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.e.j.s.Server Started @2461878ms
   [junit4]   2> 2461695 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:45727/hdfs__lucene2-us-west.apache.org_45727__home_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.7_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001_tempDir-002_control_data, replicaType=NRT, hostContext=/y_xqp/bj, hostPort=37038, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001/control-001/cores}
   [junit4]   2> 2461695 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2461695 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2461695 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.2
   [junit4]   2> 2461696 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2461696 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2461696 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-24T02:53:39.165Z
   [junit4]   2> 2461749 INFO  (zkConnectionManagerCallback-4349-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2461750 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2461750 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001/control-001/solr.xml
   [junit4]   2> 2461753 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2461753 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2461755 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@24329d4f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2462001 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2462030 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42598/solr
   [junit4]   2> 2462043 INFO  (zkConnectionManagerCallback-4353-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2462089 INFO  (zkConnectionManagerCallback-4355-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2462473 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37038_y_xqp%2Fbj
   [junit4]   2> 2462511 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.c.Overseer Overseer (id=74744820346060804-127.0.0.1:37038_y_xqp%2Fbj-n_0000000000) starting
   [junit4]   2> 2462562 INFO  (zkConnectionManagerCallback-4362-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2462564 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42598/solr ready
   [junit4]   2> 2462564 INFO  (OverseerStateUpdate-74744820346060804-127.0.0.1:37038_y_xqp%2Fbj-n_0000000000) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37038_y_xqp%2Fbj
   [junit4]   2> 2462565 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37038_y_xqp%2Fbj
   [junit4]   2> 2462566 INFO  (zkCallback-4354-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2462584 INFO  (zkCallback-4361-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2462656 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2462724 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@24329d4f
   [junit4]   2> 2462775 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@24329d4f
   [junit4]   2> 2462775 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@24329d4f
   [junit4]   2> 2462777 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001/control-001/cores
   [junit4]   2> 2462847 INFO  (zkConnectionManagerCallback-4368-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2462860 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2462860 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42598/solr ready
   [junit4]   2> 2462862 INFO  (qtp6492755-16982) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:37038_y_xqp%252Fbj&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2462864 INFO  (OverseerThreadFactory-5118-thread-1-processing-n:127.0.0.1:37038_y_xqp%2Fbj) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 2462992 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2463009 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2464057 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.2
   [junit4]   2> 2464142 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2464781 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2464836 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 2464836 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@24329d4f
   [junit4]   2> 2464836 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:45727/solr_hdfs_home
   [junit4]   2> 2464836 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 2464837 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[hdfs://lucene2-us-west.apache.org:45727/solr_hdfs_home/control_collection/core_node2/data/]
   [junit4]   2> 2464838 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:45727/solr_hdfs_home/control_collection/core_node2/data/snapshot_metadata
   [junit4]   2> 2465043 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 2465044 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 2465044 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 2465180 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 2465182 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:45727/solr_hdfs_home/control_collection/core_node2/data
   [junit4]   2> 2465269 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:45727/solr_hdfs_home/control_collection/core_node2/data/index
   [junit4]   2> 2465287 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 2465287 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 2465287 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 2465356 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 2465357 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.6650390625, noCFSRatio=0.0]
   [junit4]   2> 2465561 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37081 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-94815bc6-fdbd-4292-aa0d-90dd9a1500d8:NORMAL:127.0.0.1:36782|RBW], ReplicaUC[[DISK]DS-3dbf43d4-deb6-4417-b83d-8b214fddfd22:NORMAL:127.0.0.1:37081|RBW]]} size 69
   [junit4]   2> 2465561 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36782 is added to blk_1073741825_1001 size 69
   [junit4]   2> 2466027 WARN  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2466367 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 2466367 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2466367 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 2466496 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2466496 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2466514 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=1.2890625, floorSegmentMB=0.3115234375, forceMergeDeletesPctAllowed=18.773273664349325, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6181227540243732, deletesPctAllowed=21.91533359046002
   [junit4]   2> 2466660 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2bf93efe[control_collection_shard1_replica_n1] main]
   [junit4]   2> 2466661 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2466661 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2466662 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2466664 WARN  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.h.HdfsLocalityReporter Could not retrieve locality information for hdfs://lucene2-us-west.apache.org:41367/solr3 due to exception: java.net.ConnectException: Call From lucene2-us-west.apache.org/127.0.0.1 to lucene2-us-west.apache.org:41367 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
   [junit4]   2> 2466664 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1634380204351684608
   [junit4]   2> 2466720 INFO  (searcherExecutor-5123-thread-1-processing-n:127.0.0.1:37038_y_xqp%2Fbj x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@2bf93efe[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2466729 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 2466729 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/control_collection/leaders/shard1
   [junit4]   2> 2466770 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2466770 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2466770 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:37038/y_xqp/bj/control_collection_shard1_replica_n1/
   [junit4]   2> 2466770 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2466770 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:37038/y_xqp/bj/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 2466770 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2466770 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/74744820346060804-core_node2-n_0000000000
   [junit4]   2> 2466789 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:37038/y_xqp/bj/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 2466839 INFO  (zkCallback-4354-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2466839 INFO  (zkCallback-4354-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2466877 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2466935 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3943
   [junit4]   2> 2466999 INFO  (qtp6492755-16982) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 2467051 INFO  (zkCallback-4354-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2467051 INFO  (zkCallback-4354-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2467051 INFO  (zkCallback-4354-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2467052 INFO  (qtp6492755-16982) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:37038_y_xqp%252Fbj&wt=javabin&version=2} status=0 QTime=4190
   [junit4]   2> 2467171 INFO  (zkConnectionManagerCallback-4373-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2467172 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2467191 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42598/solr ready
   [junit4]   2> 2467191 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2467192 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2467307 INFO  (OverseerThreadFactory-5118-thread-2-processing-n:127.0.0.1:37038_y_xqp%2Fbj) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 2467335 INFO  (OverseerCollectionConfigSetProcessor-74744820346060804-127.0.0.1:37038_y_xqp%2Fbj-n_0000000000) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2467643 WARN  (OverseerThreadFactory-5118-thread-2-processing-n:127.0.0.1:37038_y_xqp%2Fbj) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 2467645 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 2467645 INFO  (qtp6492755-16984) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=452
   [junit4]   2> 2467663 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating jetty instances pullReplicaCount=0 numOtherReplicas=4
   [junit4]   2> 2468321 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001/shard-1-001 of type NRT
   [junit4]   2> 2468336 WARN  (closeThreadPool-4374-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2468336 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2468336 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2468336 INFO  (closeThreadPool-4374-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2468398 INFO  (closeThreadPool-4374-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2468398 INFO  (closeThreadPool-4374-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2468398 INFO  (closeThreadPool-4374-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2468409 INFO  (closeThreadPool-4374-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3950310a{/y_xqp/bj,null,AVAILABLE}
   [junit4]   2> 2468409 INFO  (closeThreadPool-4374-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4823569d{HTTP/1.1,[http/1.1]}{127.0.0.1:38949}
   [junit4]   2> 2468409 INFO  (closeThreadPool-4374-thread-1) [    ] o.e.j.s.Server Started @2468592ms
   [junit4]   2> 2468409 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:45727/hdfs__lucene2-us-west.apache.org_45727__home_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.7_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/y_xqp/bj, hostPort=38949, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001/shard-1-001/cores}
   [junit4]   2> 2468409 ERROR (closeThreadPool-4374-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2468409 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2468409 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.2
   [junit4]   2> 2468410 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2468410 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2468410 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-24T02:53:45.879Z
   [junit4]   2> 2468411 INFO  (zkConnectionManagerCallback-4376-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2468412 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2468412 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001/shard-1-001/solr.xml
   [junit4]   2> 2468415 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2468415 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2468433 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@24329d4f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2468581 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2468669 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42598/solr
   [junit4]   2> 2468785 INFO  (zkConnectionManagerCallback-4380-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2468858 INFO  (zkConnectionManagerCallback-4382-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2468861 INFO  (closeThreadPool-4374-thread-1) [n:127.0.0.1:38949_y_xqp%2Fbj    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2468880 INFO  (closeThreadPool-4374-thread-1) [n:127.0.0.1:38949_y_xqp%2Fbj    ] o.a.s.c.ZkController Publish node=127.0.0.1:38949_y_xqp%2Fbj as DOWN
   [junit4]   2> 2468881 INFO  (closeThreadPool-4374-thread-1) [n:127.0.0.1:38949_y_xqp%2Fbj    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2468881 INFO  (closeThreadPool-4374-thread-1) [n:127.0.0.1:38949_y_xqp%2Fbj    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38949_y_xqp%2Fbj
   [junit4]   2> 2468895 INFO  (zkCallback-4361-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2468931 INFO  (zkCallback-4381-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2468931 INFO  (zkCallback-4372-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2468947 INFO  (zkCallback-4354-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2469053 INFO  (zkConnectionManagerCallback-4389-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2469054 INFO  (closeThreadPool-4374-thread-1) [n:127.0.0.1:38949_y_xqp%2Fbj    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2469055 INFO  (closeThreadPool-4374-thread-1) [n:127.0.0.1:38949_y_xqp%2Fbj    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42598/solr ready
   [junit4]   2> 2469055 INFO  (closeThreadPool-4374-thread-1) [n:127.0.0.1:38949_y_xqp%2Fbj    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2469139 INFO  (closeThreadPool-4374-thread-1) [n:127.0.0.1:38949_y_xqp%2Fbj    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@24329d4f
   [junit4]   2> 2469177 INFO  (closeThreadPool-4374-thread-1) [n:127.0.0.1:38949_y_xqp%2Fbj    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@24329d4f
   [junit4]   2> 2469177 INFO  (closeThreadPool-4374-thread-1) [n:127.0.0.1:38949_y_xqp%2Fbj    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@24329d4f
   [junit4]   2> 2469190 INFO  (closeThreadPool-4374-thread-1) [n:127.0.0.1:38949_y_xqp%2Fbj    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001/shard-1-001/cores
   [junit4]   2> 2469398 INFO  (OverseerCollectionConfigSetProcessor-74744820346060804-127.0.0.1:37038_y_xqp%2Fbj-n_0000000000) [n:127.0.0.1:37038_y_xqp%2Fbj    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2469485 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001/shard-2-001 of type NRT
   [junit4]   2> 2469485 WARN  (closeThreadPool-4374-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2469485 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2469485 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2469485 INFO  (closeThreadPool-4374-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2469592 INFO  (closeThreadPool-4374-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2469592 INFO  (closeThreadPool-4374-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2469592 INFO  (closeThreadPool-4374-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2469593 INFO  (closeThreadPool-4374-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@f561c32{/y_xqp/bj,null,AVAILABLE}
   [junit4]   2> 2469593 INFO  (closeThreadPool-4374-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6c5a9eeb{HTTP/1.1,[http/1.1]}{127.0.0.1:45889}
   [junit4]   2> 2469593 INFO  (closeThreadPool-4374-thread-1) [    ] o.e.j.s.Server Started @2469776ms
   [junit4]   2> 2469593 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:45727/hdfs__lucene2-us-west.apache.org_45727__home_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.7_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001_tempDir-002_jetty2, solrconfig=solrconfig.xml, hostContext=/y_xqp/bj, hostPort=45889, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001/shard-2-001/cores}
   [junit4]   2> 2469594 ERROR (closeThreadPool-4374-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2469594 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2469594 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.2
   [junit4]   2> 2469594 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2469594 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2469594 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-24T02:53:47.063Z
   [junit4]   2> 2469628 INFO  (zkConnectionManagerCallback-4392-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2469649 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2469649 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001/shard-2-001/solr.xml
   [junit4]   2> 2469652 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2469652 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2469686 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@24329d4f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2471414 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2471414 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[71DC18D72A25C854]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_71DC18D72A25C854-001/shard-3-001 of type NRT
   [junit4]   2> 2471455 WARN  (closeThreadPool-4374-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2471594 INFO  (closeThreadPool-4374-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42598/solr
   [junit4]   2> 2471690 INFO  (zkConnectionManagerCallback-4396-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2471727 INFO  (closeThreadPool-4374-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2471727 INFO  (closeThreadPool-4374-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2471727 INFO  (closeThreadPool-4374-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2471856 INFO  (closeThreadPool-4374-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2471856 INFO  (closeThreadPool-4374-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2471856 INFO  (closeThreadPool-4374-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2471857 INFO  (closeThreadPool-4374-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6fbf1d6d{/y_xqp/bj,null,AVAILABLE}
   [junit4]   2> 2471873 INFO  (closeThreadPool-4374-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@613ac6e4{HTTP/1.1,[http/1.1]}{127.0.0.1:33487}
   [junit4]   2> 2471873 INFO  (closeThreadPool-4374-thread-2) [    ] o.e.j.s.Server Started @2472056ms
   [junit4]   2> 2471873 INFO  (closeThreadPool-4374-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:45727/hdfs__lucene2-us-west.apache.org_45727__home_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.7_checkou

[...truncated too long message...]

rce] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1462283539
     [copy] Copying 242 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1462283539
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1462283539

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 84ms :: artifacts dl 3ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 493 minutes 23 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1072)
                at hudson.FilePath.act(FilePath.java:1061)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1835)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1074)
        at hudson.FilePath.act(FilePath.java:1061)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 18 - Still Unstable

Dawid Weiss-2
In reply to this post by Jan Høydahl / Cominvent
Well, I kind of think anything related to indexing/ merging would be
qualified as "serious" or at least "serious-ish". ;) I know there's
been a few changes to the IndexWriter lately, so I thought it might
have been a consequence of that -- if you think it's not new then it's
probably something else. I did not investigate; I just look at jenkins
logs from time to time to see if there's anything important in there.

Dawid

On Thu, May 23, 2019 at 10:24 PM Jan Høydahl <[hidden email]> wrote:

>
> I see this test failing now and then for the last year.
> I have not studied the stack traces, can you say a few more words about what you found and why it may be serious?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> 23. mai 2019 kl. 12:29 skrev Dawid Weiss <[hidden email]>:
>
> Hmm... this looks serious, no?
> D.
>
> On Thu, May 23, 2019 at 2:27 AM Apache Jenkins Server
> <[hidden email]> wrote:
>
>
> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.7/18/
>
> 1 tests failed.
> FAILED:  org.apache.lucene.index.TestConcurrentMergeScheduler.testFlushExceptions
>
> Error Message:
>
>
> Stack Trace:
> java.lang.AssertionError
>        at __randomizedtesting.SeedInfo.seed([730AC7E1D88016C6:C760943BD16080F2]:0)
>        at org.junit.Assert.fail(Assert.java:86)
>        at org.junit.Assert.assertTrue(Assert.java:41)
>        at org.junit.Assert.assertTrue(Assert.java:52)
>        at org.apache.lucene.index.TestConcurrentMergeScheduler.testFlushExceptions(TestConcurrentMergeScheduler.java:128)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:498)
>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
>        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
>        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>        at java.lang.Thread.run(Thread.java:748)
>
>
>
>
> Build Log:
> [...truncated 991 lines...]
>   [junit4] Suite: org.apache.lucene.index.TestConcurrentMergeScheduler
>   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
>   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestConcurrentMergeScheduler -Dtests.method=testFlushExceptions -Dtests.seed=730AC7E1D88016C6 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/test-data/enwiki.random.lines.txt -Dtests.locale=hr -Dtests.timezone=Asia/Pyongyang -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
>   [junit4] FAILURE 0.10s J0 | TestConcurrentMergeScheduler.testFlushExceptions <<<
>   [junit4]    > Throwable #1: java.lang.AssertionError
>   [junit4]    >        at __randomizedtesting.SeedInfo.seed([730AC7E1D88016C6:C760943BD16080F2]:0)
>   [junit4]    >        at org.apache.lucene.index.TestConcurrentMergeScheduler.testFlushExceptions(TestConcurrentMergeScheduler.java:128)
>   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
>   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/core/test/J0/temp/lucene.index.TestConcurrentMergeScheduler_730AC7E1D88016C6-001
>   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {field=PostingsFormat(name=Memory), id=PostingsFormat(name=LuceneFixedGap), content=Lucene50(blocksize=128)}, docValues:{}, maxPointsInLeafNode=1671, maxMBSortInHeap=6.217899396932761, sim=RandomSimilarity(queryNorm=false): {content=IB LL-DZ(0.3)}, locale=hr, timezone=Asia/Pyongyang
>   [junit4]   2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=92748976,total=534773760
>   [junit4]   2> NOTE: All tests run in this JVM: [TestExceedMaxTermLength, TestDocumentsWriterStallControl, TestDirectory, TestPersistentSnapshotDeletionPolicy, TestNumericDocValuesUpdates, Test4GBStoredFields, TestRegexpRandom, TestLongRangeFieldQueries, TestToken, TestMergedIterator, TestSimilarityProvider, TestMixedCodecs, TestTimSorterWorstCase, TestBinaryDocValuesUpdates, TestConcurrentMergeScheduler]
>   [junit4] Completed [193/488 (1!)] on J0 in 22.42s, 16 tests, 1 failure <<< FAILURES!
>
> [...truncated 64016 lines...]
> [asciidoctor:convert] asciidoctor: ERROR: about-this-guide.adoc: line 1: invalid part, must have at least one section (e.g., chapter, appendix, etc.)
> [asciidoctor:convert] asciidoctor: ERROR: solr-glossary.adoc: line 1: invalid part, must have at least one section (e.g., chapter, appendix, etc.)
>     [java] Processed 2465 links (2015 relative) to 3226 anchors in 246 files
>     [echo] Validated Links & Anchors via: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-ref-guide/bare-bones-html/
>
> -documentation-lint:
>    [jtidy] Checking for broken html (such as invalid tags)...
>   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/jtidy_tmp
>     [echo] Checking for broken links...
>     [exec]
>     [exec] Crawl/parse...
>     [exec]
>     [exec] Verify...
>     [echo] Checking for malformed docs...
>
> jar-checksums:
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> common.resolve:
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> jar-checksums:
>    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/null1636655966
>     [copy] Copying 36 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/null1636655966
>   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/null1636655966
>
> resolve-example:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> resolve-server:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> ivy-availability-check:
> [loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
>
> -ivy-fail-disallowed-ivy-version:
>
> ivy-fail:
>
> ivy-fail:
>
> ivy-configure:
> [ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml
>
> resolve:
>
> jar-checksums:
>    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1210438799
>     [copy] Copying 242 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1210438799
>   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null1210438799
>
> check-working-copy:
> [ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
> [ivy:cachepath]         confs: [default]
> [ivy:cachepath]         found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
> [ivy:cachepath]         found com.jcraft#jsch;0.1.53 in public
> [ivy:cachepath]         found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
> [ivy:cachepath]         found org.apache.httpcomponents#httpclient;4.3.6 in public
> [ivy:cachepath]         found org.apache.httpcomponents#httpcore;4.3.3 in public
> [ivy:cachepath]         found commons-logging#commons-logging;1.1.3 in public
> [ivy:cachepath]         found commons-codec#commons-codec;1.6 in public
> [ivy:cachepath]         found org.slf4j#slf4j-api;1.7.2 in public
> [ivy:cachepath] :: resolution report :: resolve 53ms :: artifacts dl 8ms
>        ---------------------------------------------------------------------
>        |                  |            modules            ||   artifacts   |
>        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
>        ---------------------------------------------------------------------
>        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
>        ---------------------------------------------------------------------
> [wc-checker] Initializing working copy...
> [wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> [wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
> [wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
> [wc-checker] Checking working copy status...
>
> -jenkins-base:
>
> BUILD SUCCESSFUL
> Total time: 261 minutes 19 seconds
> Archiving artifacts
> java.lang.InterruptedException: no matches found within 10000
>        at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
>        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
>        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
>        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
> Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
>                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
>                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
>                at hudson.remoting.Channel.call(Channel.java:955)
>                at hudson.FilePath.act(FilePath.java:1072)
>                at hudson.FilePath.act(FilePath.java:1061)
>                at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
>                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
>                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
>                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
>                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
>                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
>                at hudson.model.Build$BuildExecution.post2(Build.java:186)
>                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
>                at hudson.model.Run.execute(Run.java:1835)
>                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>                at hudson.model.ResourceController.execute(ResourceController.java:97)
>                at hudson.model.Executor.run(Executor.java:429)
> Caused: hudson.FilePath$TunneledInterruptedException
>        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
>        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
>        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
>        at hudson.remoting.Request$2.run(Request.java:369)
>        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>        at java.lang.Thread.run(Thread.java:748)
> Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
>        at hudson.FilePath.act(FilePath.java:1074)
>        at hudson.FilePath.act(FilePath.java:1061)
>        at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
>        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
>        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
>        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
>        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
>        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
>        at hudson.model.Build$BuildExecution.post2(Build.java:186)
>        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
>        at hudson.model.Run.execute(Run.java:1835)
>        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>        at hudson.model.ResourceController.execute(ResourceController.java:97)
>        at hudson.model.Executor.run(Executor.java:429)
> No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
> Recording test results
> Build step 'Publish JUnit test result report' changed build result to UNSTABLE
> Email was triggered for: Unstable (Test Failures)
> Sending email for trigger: Unstable (Test Failures)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 20 - Failure

Apache Jenkins Server-2
In reply to this post by Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.7/20/

2 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
        at __randomizedtesting.SeedInfo.seed([8DB2B1576E8C69DC:5E68E8DC0700424]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:195)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:143)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:138)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:1008)
        at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:117)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1075)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1047)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at __randomizedtesting.SeedInfo.seed([8DB2B1576E8C69DC:BE009993633BB36B]:0)
        at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectField.<init>(DirectPostingsFormat.java:458)
        at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectFields.<init>(DirectPostingsFormat.java:129)
        at org.apache.lucene.codecs.memory.DirectPostingsFormat.fieldsProducer(DirectPostingsFormat.java:113)
        at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:283)
        at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:363)
        at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113)
        at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83)
        at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4627)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2177)
        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5139)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1619)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
        at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:189)
        at org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit(TestDocTermOrdsUninvertLimit.java:66)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)




Build Log:
[...truncated 13716 lines...]
   [junit4] Suite: org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDocTermOrdsUninvertLimit -Dtests.method=testTriggerUnInvertLimit -Dtests.seed=8DB2B1576E8C69DC -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/test-data/enwiki.random.lines.txt -Dtests.locale=ar-IQ -Dtests.timezone=America/Rosario -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR    291s J0 | TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > at __randomizedtesting.SeedInfo.seed([8DB2B1576E8C69DC:BE009993633BB36B]:0)
   [junit4]    > at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectField.<init>(DirectPostingsFormat.java:458)
   [junit4]    > at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectFields.<init>(DirectPostingsFormat.java:129)
   [junit4]    > at org.apache.lucene.codecs.memory.DirectPostingsFormat.fieldsProducer(DirectPostingsFormat.java:113)
   [junit4]    > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:283)
   [junit4]    > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:363)
   [junit4]    > at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113)
   [junit4]    > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83)
   [junit4]    > at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172)
   [junit4]    > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4627)
   [junit4]    > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
   [junit4]    > at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]    > at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2177)
   [junit4]    > at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5139)
   [junit4]    > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1619)
   [junit4]    > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
   [junit4]    > at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:189)
   [junit4]    > at org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit(TestDocTermOrdsUninvertLimit.java:66)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {field=PostingsFormat(name=Direct)}, docValues:{}, maxPointsInLeafNode=1885, maxMBSortInHeap=6.382781388324267, sim=RandomSimilarity(queryNorm=false): {field=DFR I(ne)B1}, locale=ar-IQ, timezone=America/Rosario
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=245978896,total=477626368
   [junit4]   2> NOTE: All tests run in this JVM: [TestHdfsUpdateLog, AliasIntegrationTest, DistributedTermsComponentTest, BlockCacheTest, TestElisionMultitermQuery, ResponseLogComponentTest, TestIndexSearcher, ZkNodePropsTest, AlternateDirectoryTest, TestLRUCache, TestXmlQParserPlugin, LegacyCloudClusterPropTest, TestInitParams, AssignTest, ReplaceNodeTest, LeaderFailureAfterFreshStartTest, CustomCollectionTest, NodeMarkersRegistrationTest, SystemInfoHandlerTest, TestSizeLimitedDistributedMap, SolrGraphiteReporterTest, MetricTriggerIntegrationTest, TestTolerantUpdateProcessorCloud, DistributedVersionInfoTest, MergeStrategyTest, CreateCollectionCleanupTest, TestCloudPseudoReturnFields, TestSubQueryTransformerCrossCore, IndexSchemaTest, SolrXmlInZkTest, DocValuesMissingTest, TestStressLucene, SuggestComponentContextFilterQueryTest, ShardRoutingCustomTest, CurrencyRangeFacetCloudTest, TestDocTermOrdsUninvertLimit]
   [junit4] Completed [127/842 (1!)] on J0 in 291.57s, 1 test, 1 error <<< FAILURES!

[...truncated 1231 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.StressHdfsTest
   [junit4]   2> 4116399 INFO  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/init-core-data-001
   [junit4]   2> 4116400 INFO  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 4116415 INFO  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 4116415 INFO  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 4116628 WARN  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 4116705 WARN  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 4116707 INFO  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 4116798 INFO  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_lucene2.us.west_apache_org_46393_hdfs____xbblmd/webapp
   [junit4]   2> 4118317 INFO  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.m.log Started HttpServer2$[hidden email]:46393
   [junit4]   2> 4119648 WARN  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 4119683 INFO  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 4119810 INFO  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_37837_datanode____115jfj/webapp
   [junit4]   2> 4120982 INFO  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37837
   [junit4]   2> 4121852 WARN  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 4121853 INFO  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 4122173 INFO  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_40602_datanode____ayryjn/webapp
   [junit4]   2> 4123463 ERROR (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:44066) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 4124044 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x9222eedca3508d: from storage DS-ecdaa019-2d25-4830-8fb9-8e037a314267 node DatanodeRegistration(127.0.0.1:45334, datanodeUuid=da90ac70-e627-48d4-bb5e-631aaddb776e, infoPort=34080, infoSecurePort=0, ipcPort=41230, storageInfo=lv=-56;cid=testClusterID;nsid=593142609;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 4124044 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x9222eedca3508d: from storage DS-bd758590-56ed-4c34-b2f9-5dd6d9ef1a19 node DatanodeRegistration(127.0.0.1:45334, datanodeUuid=da90ac70-e627-48d4-bb5e-631aaddb776e, infoPort=34080, infoSecurePort=0, ipcPort=41230, storageInfo=lv=-56;cid=testClusterID;nsid=593142609;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 4124474 INFO  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40602
   [junit4]   2> 4124893 ERROR (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:44066) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 4124945 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x9222ef18729a0c: from storage DS-48e37673-cd7c-4c80-8bb3-40ac7410a17c node DatanodeRegistration(127.0.0.1:38038, datanodeUuid=89dabc1c-d59e-4018-be86-851f660b9d6b, infoPort=34093, infoSecurePort=0, ipcPort=38089, storageInfo=lv=-56;cid=testClusterID;nsid=593142609;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 4124945 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x9222ef18729a0c: from storage DS-0c710ac6-4b85-4e7d-b80d-0f85def6d517 node DatanodeRegistration(127.0.0.1:38038, datanodeUuid=89dabc1c-d59e-4018-be86-851f660b9d6b, infoPort=34093, infoSecurePort=0, ipcPort=38089, storageInfo=lv=-56;cid=testClusterID;nsid=593142609;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 4125567 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 4125567 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 4125567 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 4125677 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.ZkTestServer start zk server on port:43520
   [junit4]   2> 4125677 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:43520
   [junit4]   2> 4125677 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 43520
   [junit4]   2> 4125700 INFO  (zkConnectionManagerCallback-14911-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4125756 INFO  (zkConnectionManagerCallback-14913-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4125757 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 4125758 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 4125759 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 4125760 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 4125761 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 4125761 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 4125774 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 4125775 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 4125776 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 4125777 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 4125778 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 4125791 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 4126345 WARN  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 4126346 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 4126346 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 4126346 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 4126347 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4126347 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4126347 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 4126347 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@51901a3c{/,null,AVAILABLE}
   [junit4]   2> 4126348 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4af97cce{HTTP/1.1,[http/1.1]}{127.0.0.1:35087}
   [junit4]   2> 4126348 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.e.j.s.Server Started @4126384ms
   [junit4]   2> 4126348 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:44066/hdfs__lucene2-us-west.apache.org_44066__home_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.7_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001_tempDir-002_control_data, replicaType=NRT, hostContext=/, hostPort=35087, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/control-001/cores}
   [junit4]   2> 4126348 ERROR (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4126348 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4126348 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.2
   [junit4]   2> 4126348 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4126348 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4126348 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-25T14:59:17.368Z
   [junit4]   2> 4126362 INFO  (zkConnectionManagerCallback-14915-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4126363 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 4126363 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/control-001/solr.xml
   [junit4]   2> 4126379 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4126379 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4126380 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4127146 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 4127167 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43520/solr
   [junit4]   2> 4127180 INFO  (zkConnectionManagerCallback-14919-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4127181 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10993c354610003, likely client has closed socket
   [junit4]   2> 4127182 INFO  (zkConnectionManagerCallback-14921-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4127450 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [n:127.0.0.1:35087_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35087_
   [junit4]   2> 4127450 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [n:127.0.0.1:35087_    ] o.a.s.c.Overseer Overseer (id=74753335971872772-127.0.0.1:35087_-n_0000000000) starting
   [junit4]   2> 4127470 INFO  (zkConnectionManagerCallback-14928-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4127488 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [n:127.0.0.1:35087_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43520/solr ready
   [junit4]   2> 4127491 INFO  (OverseerStateUpdate-74753335971872772-127.0.0.1:35087_-n_0000000000) [n:127.0.0.1:35087_    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:35087_
   [junit4]   2> 4127493 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [n:127.0.0.1:35087_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35087_
   [junit4]   2> 4127512 INFO  (zkCallback-14927-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4127540 INFO  (zkCallback-14920-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4127607 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [n:127.0.0.1:35087_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4127695 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [n:127.0.0.1:35087_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580
   [junit4]   2> 4127732 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [n:127.0.0.1:35087_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580
   [junit4]   2> 4127732 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [n:127.0.0.1:35087_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580
   [junit4]   2> 4127733 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [n:127.0.0.1:35087_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/control-001/cores
   [junit4]   2> 4127887 INFO  (zkConnectionManagerCallback-14934-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4127888 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4127889 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43520/solr ready
   [junit4]   2> 4127907 INFO  (qtp1116618231-45928) [n:127.0.0.1:35087_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:35087_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4127922 INFO  (OverseerThreadFactory-8325-thread-1-processing-n:127.0.0.1:35087_) [n:127.0.0.1:35087_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 4128028 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 4128028 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4129167 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.2
   [junit4]   2> 4129261 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 4130021 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 4130145 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 4130162 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580
   [junit4]   2> 4130163 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:44066/solr_hdfs_home
   [junit4]   2> 4130163 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 4130163 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[hdfs://lucene2-us-west.apache.org:44066/solr_hdfs_home/control_collection/core_node2/data/]
   [junit4]   2> 4130164 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:44066/solr_hdfs_home/control_collection/core_node2/data/snapshot_metadata
   [junit4]   2> 4130200 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 4130200 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 4130200 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 4130257 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 4130258 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:44066/solr_hdfs_home/control_collection/core_node2/data
   [junit4]   2> 4130328 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:44066/solr_hdfs_home/control_collection/core_node2/data/index
   [junit4]   2> 4130346 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 4130346 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 4130346 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 4130389 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 4130389 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=12.8525390625, floorSegmentMB=0.783203125, forceMergeDeletesPctAllowed=0.1856908761435505, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3258451717769235, deletesPctAllowed=33.82998121166678
   [junit4]   2> 4130490 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45334 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-48e37673-cd7c-4c80-8bb3-40ac7410a17c:NORMAL:127.0.0.1:38038|RBW], ReplicaUC[[DISK]DS-bd758590-56ed-4c34-b2f9-5dd6d9ef1a19:NORMAL:127.0.0.1:45334|RBW]]} size 69
   [junit4]   2> 4130490 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38038 is added to blk_1073741825_1001 size 69
   [junit4]   2> 4131022 WARN  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 4131274 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 4131275 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4131275 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 4131335 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4131335 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4131349 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=49, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 4131423 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5de16850[control_collection_shard1_replica_n1] main]
   [junit4]   2> 4131424 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 4131425 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 4131425 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 4131458 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1634516454581731328
   [junit4]   2> 4131461 INFO  (searcherExecutor-8330-thread-1-processing-n:127.0.0.1:35087_ x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@5de16850[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4131498 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 4131498 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/control_collection/leaders/shard1
   [junit4]   2> 4131512 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 4131512 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 4131512 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:35087/control_collection_shard1_replica_n1/
   [junit4]   2> 4131513 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 4131513 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:35087/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 4131513 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 4131513 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/74753335971872772-core_node2-n_0000000000
   [junit4]   2> 4131622 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:35087/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 4131642 INFO  (zkCallback-14920-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4131643 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 4131653 INFO  (zkCallback-14920-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4131677 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3649
   [junit4]   2> 4131783 INFO  (qtp1116618231-45928) [n:127.0.0.1:35087_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 4131832 INFO  (zkCallback-14920-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4131832 INFO  (zkCallback-14920-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4131833 INFO  (qtp1116618231-45928) [n:127.0.0.1:35087_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:35087_&wt=javabin&version=2} status=0 QTime=3926
   [junit4]   2> 4131833 INFO  (zkCallback-14920-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4131904 INFO  (zkConnectionManagerCallback-14939-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4131905 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4131906 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43520/solr ready
   [junit4]   2> 4131906 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 4131907 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4131962 INFO  (OverseerCollectionConfigSetProcessor-74753335971872772-127.0.0.1:35087_-n_0000000000) [n:127.0.0.1:35087_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 4131962 INFO  (OverseerThreadFactory-8325-thread-2-processing-n:127.0.0.1:35087_) [n:127.0.0.1:35087_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 4132184 WARN  (OverseerThreadFactory-8325-thread-2-processing-n:127.0.0.1:35087_) [n:127.0.0.1:35087_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 4132186 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 4132186 INFO  (qtp1116618231-45930) [n:127.0.0.1:35087_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=279
   [junit4]   2> 4132196 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating jetty instances pullReplicaCount=0 numOtherReplicas=7
   [junit4]   2> 4132699 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-1-001 of type NRT
   [junit4]   2> 4132718 WARN  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 4132719 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 4132719 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 4132719 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 4132844 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4132844 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4132844 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 4132847 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2fded126{/,null,AVAILABLE}
   [junit4]   2> 4132849 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@13034481{HTTP/1.1,[http/1.1]}{127.0.0.1:44517}
   [junit4]   2> 4132849 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.Server Started @4132886ms
   [junit4]   2> 4132849 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:44066/hdfs__lucene2-us-west.apache.org_44066__home_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.7_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001_tempDir-002_jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=44517, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-1-001/cores}
   [junit4]   2> 4132850 ERROR (closeThreadPool-14940-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4132850 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4132850 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.2
   [junit4]   2> 4132850 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4132850 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4132850 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-25T14:59:23.870Z
   [junit4]   2> 4132933 INFO  (zkConnectionManagerCallback-14942-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4132990 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 4132990 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-1-001/solr.xml
   [junit4]   2> 4132993 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10993c354610008, likely client has closed socket
   [junit4]   2> 4133010 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4133010 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4133012 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4133400 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 4133425 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43520/solr
   [junit4]   2> 4133426 INFO  (zkConnectionManagerCallback-14946-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4133513 INFO  (zkConnectionManagerCallback-14948-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4133586 INFO  (closeThreadPool-14940-thread-1) [n:127.0.0.1:44517_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4133601 INFO  (closeThreadPool-14940-thread-1) [n:127.0.0.1:44517_    ] o.a.s.c.ZkController Publish node=127.0.0.1:44517_ as DOWN
   [junit4]   2> 4133602 INFO  (closeThreadPool-14940-thread-1) [n:127.0.0.1:44517_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4133602 INFO  (closeThreadPool-14940-thread-1) [n:127.0.0.1:44517_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44517_
   [junit4]   2> 4133603 INFO  (zkCallback-14920-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4133622 INFO  (zkCallback-14938-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4133638 INFO  (zkCallback-14927-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4133638 INFO  (zkCallback-14947-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4133733 INFO  (zkConnectionManagerCallback-14955-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4133734 INFO  (closeThreadPool-14940-thread-1) [n:127.0.0.1:44517_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 4133768 INFO  (closeThreadPool-14940-thread-1) [n:127.0.0.1:44517_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43520/solr ready
   [junit4]   2> 4133768 INFO  (closeThreadPool-14940-thread-1) [n:127.0.0.1:44517_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4133866 INFO  (closeThreadPool-14940-thread-1) [n:127.0.0.1:44517_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580
   [junit4]   2> 4133904 INFO  (closeThreadPool-14940-thread-1) [n:127.0.0.1:44517_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580
   [junit4]   2> 4133904 INFO  (closeThreadPool-14940-thread-1) [n:127.0.0.1:44517_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580
   [junit4]   2> 4133922 INFO  (closeThreadPool-14940-thread-1) [n:127.0.0.1:44517_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-1-001/cores
   [junit4]   2> 4134063 INFO  (OverseerCollectionConfigSetProcessor-74753335971872772-127.0.0.1:35087_-n_0000000000) [n:127.0.0.1:35087_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 4134239 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-2-001 of type NRT
   [junit4]   2> 4134249 WARN  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 4134250 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 4134250 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 4134250 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 4134270 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4134270 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4134270 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 4134270 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2f2b59b{/,null,AVAILABLE}
   [junit4]   2> 4134271 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@46441701{HTTP/1.1,[http/1.1]}{127.0.0.1:36152}
   [junit4]   2> 4134271 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.Server Started @4134307ms
   [junit4]   2> 4134271 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:44066/hdfs__lucene2-us-west.apache.org_44066__home_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.7_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001_tempDir-002_jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=36152, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-2-001/cores}
   [junit4]   2> 4134271 ERROR (closeThreadPool-14940-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4134271 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4134271 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.2
   [junit4]   2> 4134271 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4134271 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4134271 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-25T14:59:25.291Z
   [junit4]   2> 4134273 INFO  (zkConnectionManagerCallback-14958-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4134306 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 4134306 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-2-001/solr.xml
   [junit4]   2> 4134325 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4134325 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4134344 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4134929 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 4135064 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43520/solr
   [junit4]   2> 4135066 INFO  (zkConnectionManagerCallback-14962-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4135117 INFO  (zkConnectionManagerCallback-14964-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4135161 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 4135164 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.ZkController Publish node=127.0.0.1:36152_ as DOWN
   [junit4]   2> 4135181 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4135181 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36152_
   [junit4]   2> 4135183 INFO  (zkCallback-14920-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4135183 INFO  (zkCallback-14938-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4135197 INFO  (zkCallback-14947-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4135205 INFO  (zkCallback-14927-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4135241 INFO  (zkCallback-14963-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4135335 INFO  (zkCallback-14954-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4135404 INFO  (zkConnectionManagerCallback-14971-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4135436 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 4135437 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43520/solr ready
   [junit4]   2> 4135509 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4135579 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580
   [junit4]   2> 4135621 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580
   [junit4]   2> 4135621 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580
   [junit4]   2> 4135636 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-2-001/cores
   [junit4]   2> 4135851 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-3-001 of type NRT
   [junit4]   2> 4135878 WARN  (closeThreadPool-14940-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 4135878 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 4135878 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 4135878 INFO  (closeThreadPool-14940-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 4135898 INFO  (closeThreadPool-14940-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4135898 INFO  (closeThreadPool-14940-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4135898 INFO  (closeThreadPool-14940-thread-2) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 4135899 INFO  (closeThreadPool-14940-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@42983086{/,null,AVAILABLE}
   [junit4]   2> 4135899 INFO  (closeThreadPool-14940-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1ddbb34{HTTP/1.1,[http/1.1]}{127.0.0.1:40360}
   [junit4]   2> 4135899 INFO  (closeThreadPool-14940-thread-2) [    ] o.e.j.s.Server Started @4135936ms
   [junit4]   2> 4135899 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:44066/hdfs__lucene2-us-west.apache.org_44066__home_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.7_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001_tempDir-002_jetty3, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=40360, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-3-001/cores}
   [junit4]   2> 4135899 ERROR (closeThreadPool-14940-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4135899 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4135899 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.2
   [junit4]   2> 4135899 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4135900 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4135900 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-25T14:59:26.920Z
   [junit4]   2> 4135901 INFO  (zkConnectionManagerCallback-14974-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4135934 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 4135934 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-3-001/solr.xml
   [junit4]   2> 4135942 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4135942 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4135975 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4136706 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 4136733 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43520/solr
   [junit4]   2> 4136785 INFO  (zkConnectionManagerCallback-14978-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4136846 INFO  (zkConnectionManagerCallback-14980-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4136861 INFO  (closeThreadPool-14940-thread-2) [n:127.0.0.1:40360_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 4136969 INFO  (closeThreadPool-14940-thread-2) [n:127.0.0.1:40360_    ] o.a.s.c.ZkController Publish node=127.0.0.1:40360_ as DOWN
   [junit4]   2> 4136970 INFO  (closeThreadPool-14940-thread-2) [n:127.0.0.1:40360_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4136970 INFO  (closeThreadPool-14940-thread-2) [n:127.0.0.1:40360_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40360_
   [junit4]   2> 4136971 INFO  (zkCallback-14927-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4136992 INFO  (zkCallback-14938-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4136992 INFO  (zkCallback-14947-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4137004 INFO  (zkCallback-14954-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4137016 INFO  (zkCallback-14963-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4137016 INFO  (zkCallback-14920-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4137024 INFO  (zkCallback-14979-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4137048 INFO  (zkCallback-14970-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4137198 INFO  (zkConnectionManagerCallback-14987-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4137207 INFO  (closeThreadPool-14940-thread-2) [n:127.0.0.1:40360_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 4137208 INFO  (closeThreadPool-14940-thread-2) [n:127.0.0.1:40360_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43520/solr ready
   [junit4]   2> 4137243 INFO  (closeThreadPool-14940-thread-2) [n:127.0.0.1:40360_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4137262 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 4 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-4-001 of type NRT
   [junit4]   2> 4137262 WARN  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 4137263 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 4137263 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 4137263 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 4137316 INFO  (closeThreadPool-14940-thread-2) [n:127.0.0.1:40360_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580
   [junit4]   2> 4137354 INFO  (closeThreadPool-14940-thread-2) [n:127.0.0.1:40360_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580
   [junit4]   2> 4137354 INFO  (closeThreadPool-14940-thread-2) [n:127.0.0.1:40360_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580
   [junit4]   2> 4137368 INFO  (closeThreadPool-14940-thread-2) [n:127.0.0.1:40360_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-3-001/cores
   [junit4]   2> 4137462 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4137462 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4137462 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 4137535 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@634629fe{/,null,AVAILABLE}
   [junit4]   2> 4137652 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1b304189{HTTP/1.1,[http/1.1]}{127.0.0.1:35200}
   [junit4]   2> 4137652 INFO  (closeThreadPool-14940-thread-1) [    ] o.e.j.s.Server Started @4137689ms
   [junit4]   2> 4137652 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:44066/hdfs__lucene2-us-west.apache.org_44066__home_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.7_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001_tempDir-002_jetty4, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=35200, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-4-001/cores}
   [junit4]   2> 4137652 ERROR (closeThreadPool-14940-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4137652 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4137652 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.2
   [junit4]   2> 4137653 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4137653 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4137653 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-25T14:59:28.673Z
   [junit4]   2> 4137725 INFO  (zkConnectionManagerCallback-14990-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4137762 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 4137762 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-4-001/solr.xml
   [junit4]   2> 4137765 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4137765 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4137783 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4138294 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 4138330 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43520/solr
   [junit4]   2> 4138397 INFO  (zkConnectionManagerCallback-14994-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4138563 INFO  (zkConnectionManagerCallback-14996-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4138586 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 4138605 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.ZkController Publish node=127.0.0.1:35200_ as DOWN
   [junit4]   2> 4138605 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4138605 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35200_
   [junit4]   2> 4138607 INFO  (zkCallback-14954-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 4138607 INFO  (zkCallback-14970-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 4138607 INFO  (zkCallback-14963-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 4138607 INFO  (zkCallback-14947-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 4138607 INFO  (zkCallback-14979-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 4138628 INFO  (zkCallback-14927-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 4138628 INFO  (zkCallback-14938-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 4138628 INFO  (zkCallback-14920-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 4138629 INFO  (zkCallback-14986-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 4138646 INFO  (zkCallback-14995-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 4138678 INFO  (TEST-StressHdfsTest.test-seed#[8DB2B1576E8C69DC]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 5 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-5-001 of type NRT
   [junit4]   2> 4138678 WARN  (closeThreadPool-14940-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 4138678 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 4138678 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 4138678 INFO  (closeThreadPool-14940-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 4138790 INFO  (closeThreadPool-14940-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4138790 INFO  (closeThreadPool-14940-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4138790 INFO  (closeThreadPool-14940-thread-2) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 4138847 INFO  (closeThreadPool-14940-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@8821b9e{/,null,AVAILABLE}
   [junit4]   2> 4138883 INFO  (zkConnectionManagerCallback-15003-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4138884 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 4138885 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43520/solr ready
   [junit4]   2> 4138895 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4138899 INFO  (closeThreadPool-14940-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4465353c{HTTP/1.1,[http/1.1]}{127.0.0.1:41635}
   [junit4]   2> 4138899 INFO  (closeThreadPool-14940-thread-2) [    ] o.e.j.s.Server Started @4138936ms
   [junit4]   2> 4138899 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:44066/hdfs__lucene2-us-west.apache.org_44066__home_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.7_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001_tempDir-002_jetty5, solrconfig=solrconfig.xml, hostContext=/, hostPort=41635, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-5-001/cores}
   [junit4]   2> 4138952 ERROR (closeThreadPool-14940-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4138952 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4138952 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.2
   [junit4]   2> 4138952 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4138952 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4138952 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-25T14:59:29.972Z
   [junit4]   2> 4139006 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580
   [junit4]   2> 4139037 INFO  (zkConnectionManagerCallback-15006-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4139039 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 4139039 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-5-001/solr.xml
   [junit4]   2> 4139059 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4139059 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4139061 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4139095 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580
   [junit4]   2> 4139095 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c88580
   [junit4]   2> 4139096 INFO  (closeThreadPool-14940-thread-1) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/shard-4-001/cores
   [junit4]   2> 4139636 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 4139711 INFO  (closeThreadPool-14940-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43520/solr
   [junit4]   2> 4139732 INFO  (zkConnectionManagerCallback-15010-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4139791 INFO  (zkConnectionManagerCallback-15012-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4

[...truncated too long message...]

7s J2 | StressHdfsTest.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
   [junit4]    > at __randomizedtesting.SeedInfo.seed([8DB2B1576E8C69DC:5E68E8DC0700424]:0)
   [junit4]    > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:195)
   [junit4]    > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:143)
   [junit4]    > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:138)
   [junit4]    > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:1008)
   [junit4]    > at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:117)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1075)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1047)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 4552925 WARN  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 4552935 INFO  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 4553037 WARN  (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:44066) [    ] o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager interrupted
   [junit4]   2> 4553037 WARN  (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:44066) [    ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1388971160-127.0.0.1-1558796347568 (Datanode Uuid 89dabc1c-d59e-4018-be86-851f660b9d6b) service to lucene2-us-west.apache.org/127.0.0.1:44066
   [junit4]   2> 4553039 WARN  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 4553081 INFO  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 4553183 WARN  (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:44066) [    ] o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager interrupted
   [junit4]   2> 4553183 WARN  (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:44066) [    ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1388971160-127.0.0.1-1558796347568 (Datanode Uuid da90ac70-e627-48d4-bb5e-631aaddb776e) service to lucene2-us-west.apache.org/127.0.0.1:44066
   [junit4]   2> 4553214 INFO  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.m.log Stopped HttpServer2$[hidden email]:0
   [junit4]   2> 4553341 INFO  (SUITE-StressHdfsTest-seed#[8DB2B1576E8C69DC]-worker) [    ] o.a.s.s.h.HdfsDirectory Closing hdfs directory hdfs://lucene2-us-west.apache.org:44066/solr_hdfs_home/delete_data_dir/core_node2/data
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_8DB2B1576E8C69DC-001
   [junit4]   2> May 25, 2019 3:06:24 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 32 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=true): {}, locale=sr-Latn-ME, timezone=NZ-CHAT
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=4,free=302034976,total=524288000
   [junit4]   2> NOTE: All tests run in this JVM: [TestCustomDocTransformer, AddSchemaFieldsUpdateProcessorFactoryTest, TestSimLargeCluster, CacheHeaderTest, TestReloadDeadlock, TestNumericRangeQuery64, TestDocSet, ExitableDirectoryReaderTest, TestXmlQParser, TestDistributedStatsComponentCardinality, RangeFacetCloudTest, TermVectorComponentDistributedTest, TestCloudJSONFacetSKG, CdcrBidirectionalTest, IndexSchemaRuntimeFieldTest, TestSQLHandler, DistributedSuggestComponentTest, SmileWriterTest, TestSurroundQueryParser, NoCacheHeaderTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestSerializedLuceneMatchVersion, ExecutePlanActionTest, PreAnalyzedUpdateProcessorTest, CopyFieldTest, TestImpersonationWithHadoopAuth, HdfsRestartWhileUpdatingTest, TestCorePropertiesReload, SolrShardReporterTest, NodeAddedTriggerIntegrationTest, TestCollectionsAPIViaSolrCloudCluster, DistributedExpandComponentTest, TestManagedSchema, TestPhraseSuggestions, Tagger2Test, RecoveryAfterSoftCommitTest, TestPivotHelperCode, TemplateUpdateProcessorTest, SuggesterTest, SpatialRPTFieldTypeTest, DeleteInactiveReplicaTest, SolrGangliaReporterTest, SignificantTermsQParserPluginTest, DocValuesTest, TestSimpleQParserPlugin, SchemaApiFailureTest, TestCloudSearcherWarming, URLClassifyProcessorTest, TestBlobHandler, TestCoreBackup, SpellPossibilityIteratorTest, TestRecovery, ClassificationUpdateProcessorTest, TestLegacyTerms, TestCloudDeleteByQuery, TestFieldCacheVsDocValues, TestJavabinTupleStreamParser, ShardsWhitelistTest, TestComplexPhraseQParserPlugin, IndexBasedSpellCheckerTest, TestReplicaProperties, CoreMergeIndexesAdminHandlerTest, TestAddFieldRealTimeGet, TestSolrCloudWithDelegationTokens, TestCloudPivotFacet, SimpleFacetsTest, UpdateLogTest, TestBulkSchemaConcurrent, MetricsHistoryWithAuthIntegrationTest, TestNestedUpdateProcessor, TestDistributedGrouping, SolrJmxReporterTest, ReplicaListTransformerTest, CdcrWithNodesRestartsTest, DistributedFacetPivotLongTailTest, ShardSplitTest, TestSolrJ, TestBulkSchemaAPI, TestHashQParserPlugin, TestFastLRUCache, MultiTermTest, CdcrReplicationHandlerTest, TestDocumentBuilder, ConcurrentDeleteAndCreateCollectionTest, TestWordDelimiterFilterFactory, ZkStateWriterTest, SharedFSAutoReplicaFailoverTest, HdfsTlogReplayBufferedWhileIndexingTest, TestTestInjection, TestManagedResource, TestSolrXml, ConfigureRecoveryStrategyTest, MaxSizeAutoCommitTest, OutOfBoxZkACLAndCredentialsProvidersTest, SearchRateTriggerIntegrationTest, TestDocTermOrds, BadIndexSchemaTest, HdfsNNFailoverTest, TestNoOpRegenerator, V2StandaloneTest, ReplaceNodeNoTargetTest, TestShardHandlerFactory, SyncSliceTest, TestDistribIDF, TestManagedSynonymFilterFactory, JavabinLoaderTest, TestStreamBody, VMParamsZkACLAndCredentialsProvidersTest, TestDeleteCollectionOnDownNodes, SolrIndexMetricsTest, TestTlogReplica, TestRuleBasedAuthorizationPlugin, TestSchemaResource, TestSolrDeletionPolicy1, NodeLostTriggerTest, StressHdfsTest]
   [junit4] Completed [454/842 (2!)] on J2 in 439.72s, 1 test, 1 failure <<< FAILURES!

[...truncated 1330 lines...]
   [junit4] JVM J0: stdout was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/temp/junit4-J0-20190525_135030_8862205831055026419832.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/heapdumps/java_pid21955.hprof ...
   [junit4] Heap dump file created [462752484 bytes in 8.087 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 8924 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/build.xml:651: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/build.xml:585: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid21955.hprof

Total time: 449 minutes 46 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 21 - Still Failing

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.7/21/

2 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
shard2 is not consistent.  Got 91 from https://127.0.0.1:33145/collection1_shard2_replica_n5 (previous client) and got 92 from https://127.0.0.1:40355/collection1_shard2_replica_n1

Stack Trace:
java.lang.AssertionError: shard2 is not consistent.  Got 91 from https://127.0.0.1:33145/collection1_shard2_replica_n5 (previous client) and got 92 from https://127.0.0.1:40355/collection1_shard2_replica_n1
        at __randomizedtesting.SeedInfo.seed([6E64149722DDF104:E6302B4D8C219CFC]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1423)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1402)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:165)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1075)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1047)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test

Error Message:
Error from server at <a href="http://127.0.0.1:41798:">http://127.0.0.1:41798: ADDREPLICA failed to create replica

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at <a href="http://127.0.0.1:41798:">http://127.0.0.1:41798: ADDREPLICA failed to create replica
        at __randomizedtesting.SeedInfo.seed([6E64149722DDF104:E6302B4D8C219CFC]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.UnloadDistributedZkTest.testUnloadShardAndCollection(UnloadDistributedZkTest.java:127)
        at org.apache.solr.cloud.UnloadDistributedZkTest.test(UnloadDistributedZkTest.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1075)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1047)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14375 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> 2648209 INFO  (SUITE-ChaosMonkeySafeLeaderTest-seed#[6E64149722DDF104]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/init-core-data-001
   [junit4]   2> 2648210 WARN  (SUITE-ChaosMonkeySafeLeaderTest-seed#[6E64149722DDF104]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=88 numCloses=88
   [junit4]   2> 2648225 INFO  (SUITE-ChaosMonkeySafeLeaderTest-seed#[6E64149722DDF104]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2648227 INFO  (SUITE-ChaosMonkeySafeLeaderTest-seed#[6E64149722DDF104]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2648227 INFO  (SUITE-ChaosMonkeySafeLeaderTest-seed#[6E64149722DDF104]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 2648335 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2648354 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2648355 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2648464 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.ZkTestServer start zk server on port:34467
   [junit4]   2> 2648464 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:34467
   [junit4]   2> 2648464 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 34467
   [junit4]   2> 2648836 INFO  (zkConnectionManagerCallback-5428-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2648962 INFO  (zkConnectionManagerCallback-5430-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2648975 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2648976 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 2648976 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2648977 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2648978 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2648995 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2648995 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2648996 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2648997 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2648998 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2648998 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2649092 INFO  (zkConnectionManagerCallback-5434-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2649105 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 2650980 WARN  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2650981 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2650981 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2650981 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2651049 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2651049 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2651049 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2651053 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4fd9768a{/,null,AVAILABLE}
   [junit4]   2> 2651054 WARN  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@3cccf11[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2651055 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@17a7b529{SSL,[ssl, http/1.1]}{127.0.0.1:35182}
   [junit4]   2> 2651055 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.e.j.s.Server Started @2651140ms
   [junit4]   2> 2651055 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/tempDir-001/control/data, hostContext=/, hostPort=35182, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/control-001/cores}
   [junit4]   2> 2651055 ERROR (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2651055 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2651055 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.2
   [junit4]   2> 2651055 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2651055 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2651055 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-27T13:04:37.703Z
   [junit4]   2> 2651089 INFO  (zkConnectionManagerCallback-5436-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2651122 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2651122 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/control-001/solr.xml
   [junit4]   2> 2651125 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2651125 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2651139 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2652531 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2652615 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34467/solr
   [junit4]   2> 2652666 INFO  (zkConnectionManagerCallback-5440-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2652760 INFO  (zkConnectionManagerCallback-5442-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2653272 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35182_
   [junit4]   2> 2653272 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_    ] o.a.s.c.Overseer Overseer (id=74764209601642501-127.0.0.1:35182_-n_0000000000) starting
   [junit4]   2> 2653362 INFO  (zkConnectionManagerCallback-5449-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2653433 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34467/solr ready
   [junit4]   2> 2653469 INFO  (OverseerStateUpdate-74764209601642501-127.0.0.1:35182_-n_0000000000) [n:127.0.0.1:35182_    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:35182_
   [junit4]   2> 2653469 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35182_
   [junit4]   2> 2653505 INFO  (zkCallback-5448-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2653505 INFO  (OverseerStateUpdate-74764209601642501-127.0.0.1:35182_-n_0000000000) [n:127.0.0.1:35182_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2653671 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2653800 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
   [junit4]   2> 2653945 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
   [junit4]   2> 2653945 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
   [junit4]   2> 2654036 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/control-001/cores
   [junit4]   2> 2654306 INFO  (zkConnectionManagerCallback-5455-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2654391 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2654416 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34467/solr ready
   [junit4]   2> 2654623 INFO  (qtp1672465640-39037) [n:127.0.0.1:35182_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:35182_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2654699 INFO  (OverseerThreadFactory-4550-thread-1-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 2654896 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2654896 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2656035 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.2
   [junit4]   2> 2656124 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2656817 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 2656886 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 2656887 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
   [junit4]   2> 2656887 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 2656907 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=7, maxMergedSegmentMB=58.0263671875, floorSegmentMB=1.8046875, forceMergeDeletesPctAllowed=25.489696409831605, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=33.81642606487709
   [junit4]   2> 2656931 WARN  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2657125 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2657125 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2657147 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
   [junit4]   2> 2657147 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2657148 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@34b16866
   [junit4]   2> 2657148 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@c57fd3[control_collection_shard1_replica_n1] main]
   [junit4]   2> 2657178 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2657178 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2657179 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2657179 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1634690435726180352
   [junit4]   2> 2657183 INFO  (searcherExecutor-4555-thread-1-processing-n:127.0.0.1:35182_ x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@c57fd3[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2657200 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 2657200 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/control_collection/leaders/shard1
   [junit4]   2> 2657202 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2657202 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2657202 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:35182/control_collection_shard1_replica_n1/
   [junit4]   2> 2657202 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2657214 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:35182/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 2657214 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2657215 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/74764209601642501-core_node2-n_0000000000
   [junit4]   2> 2657217 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:35182/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 2657218 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2657252 INFO  (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2356
   [junit4]   2> 2657254 INFO  (qtp1672465640-39037) [n:127.0.0.1:35182_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 2657361 INFO  (zkCallback-5441-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2657361 INFO  (qtp1672465640-39037) [n:127.0.0.1:35182_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:35182_&wt=javabin&version=2} status=0 QTime=2738
   [junit4]   2> 2657380 INFO  (zkCallback-5441-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2657473 INFO  (zkConnectionManagerCallback-5460-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2657490 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2657491 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34467/solr ready
   [junit4]   2> 2657491 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2657526 INFO  (qtp1672465640-39037) [n:127.0.0.1:35182_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2657563 INFO  (OverseerThreadFactory-4550-thread-2-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 2657563 INFO  (OverseerCollectionConfigSetProcessor-74764209601642501-127.0.0.1:35182_-n_0000000000) [n:127.0.0.1:35182_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2657781 WARN  (OverseerThreadFactory-4550-thread-2-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 2657783 INFO  (qtp1672465640-39037) [n:127.0.0.1:35182_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 2657784 INFO  (qtp1672465640-39037) [n:127.0.0.1:35182_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=257
   [junit4]   2> 2657802 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating jetty instances pullReplicaCount=0 numOtherReplicas=3
   [junit4]   2> 2658839 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-1-001 of type NRT
   [junit4]   2> 2658876 WARN  (closeThreadPool-5461-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2658877 INFO  (closeThreadPool-5461-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2658877 INFO  (closeThreadPool-5461-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2658877 INFO  (closeThreadPool-5461-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2659130 INFO  (closeThreadPool-5461-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2659130 INFO  (closeThreadPool-5461-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2659130 INFO  (closeThreadPool-5461-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2659147 INFO  (closeThreadPool-5461-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1e2eafd4{/,null,AVAILABLE}
   [junit4]   2> 2659147 WARN  (closeThreadPool-5461-thread-1) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@38632f7b[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2659163 INFO  (closeThreadPool-5461-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@48a5d748{SSL,[ssl, http/1.1]}{127.0.0.1:33145}
   [junit4]   2> 2659163 INFO  (closeThreadPool-5461-thread-1) [    ] o.e.j.s.Server Started @2659247ms
   [junit4]   2> 2659163 INFO  (closeThreadPool-5461-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=33145, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-1-001/cores}
   [junit4]   2> 2659163 ERROR (closeThreadPool-5461-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2659163 INFO  (closeThreadPool-5461-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2659163 INFO  (closeThreadPool-5461-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.2
   [junit4]   2> 2659163 INFO  (closeThreadPool-5461-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2659163 INFO  (closeThreadPool-5461-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2659163 INFO  (closeThreadPool-5461-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-27T13:04:45.811Z
   [junit4]   2> 2659223 INFO  (zkConnectionManagerCallback-5463-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2659312 INFO  (closeThreadPool-5461-thread-1) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2659312 INFO  (closeThreadPool-5461-thread-1) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-1-001/solr.xml
   [junit4]   2> 2659316 INFO  (closeThreadPool-5461-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2659316 INFO  (closeThreadPool-5461-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2659370 INFO  (closeThreadPool-5461-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2659568 INFO  (OverseerCollectionConfigSetProcessor-74764209601642501-127.0.0.1:35182_-n_0000000000) [n:127.0.0.1:35182_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2660351 INFO  (closeThreadPool-5461-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2660462 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-2-001 of type NRT
   [junit4]   2> 2660443 INFO  (closeThreadPool-5461-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34467/solr
   [junit4]   2> 2660476 WARN  (closeThreadPool-5461-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2660476 INFO  (closeThreadPool-5461-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2660476 INFO  (closeThreadPool-5461-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2660476 INFO  (closeThreadPool-5461-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2660532 INFO  (closeThreadPool-5461-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2660532 INFO  (closeThreadPool-5461-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2660532 INFO  (closeThreadPool-5461-thread-2) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2660532 INFO  (closeThreadPool-5461-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5702f438{/,null,AVAILABLE}
   [junit4]   2> 2660533 WARN  (closeThreadPool-5461-thread-2) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@1dd1ed4c[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2660556 INFO  (closeThreadPool-5461-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@28ca1c9d{SSL,[ssl, http/1.1]}{127.0.0.1:37001}
   [junit4]   2> 2660556 INFO  (closeThreadPool-5461-thread-2) [    ] o.e.j.s.Server Started @2660641ms
   [junit4]   2> 2660556 INFO  (closeThreadPool-5461-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=37001, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-2-001/cores}
   [junit4]   2> 2660557 ERROR (closeThreadPool-5461-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2660557 INFO  (closeThreadPool-5461-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2660557 INFO  (closeThreadPool-5461-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.2
   [junit4]   2> 2660557 INFO  (closeThreadPool-5461-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2660557 INFO  (closeThreadPool-5461-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2660557 INFO  (closeThreadPool-5461-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-27T13:04:47.205Z
   [junit4]   2> 2660557 INFO  (zkConnectionManagerCallback-5467-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2660585 INFO  (zkConnectionManagerCallback-5469-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2660585 INFO  (closeThreadPool-5461-thread-2) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2660586 INFO  (closeThreadPool-5461-thread-2) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-2-001/solr.xml
   [junit4]   2> 2660621 INFO  (closeThreadPool-5461-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2660621 INFO  (closeThreadPool-5461-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2660623 INFO  (closeThreadPool-5461-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2660722 INFO  (zkConnectionManagerCallback-5471-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2660725 INFO  (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2660739 INFO  (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_    ] o.a.s.c.ZkController Publish node=127.0.0.1:33145_ as DOWN
   [junit4]   2> 2660740 INFO  (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2660740 INFO  (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33145_
   [junit4]   2> 2660741 INFO  (zkCallback-5448-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2660741 INFO  (zkCallback-5459-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2660741 INFO  (zkCallback-5441-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2660840 INFO  (zkCallback-5470-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2668154 INFO  (zkConnectionManagerCallback-5478-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2668410 INFO  (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2668411 INFO  (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34467/solr ready
   [junit4]   2> 2668503 INFO  (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2668595 INFO  (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
   [junit4]   2> 2668760 INFO  (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
   [junit4]   2> 2668761 INFO  (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
   [junit4]   2> 2668774 INFO  (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-1-001/cores
   [junit4]   2> 2668958 INFO  (closeThreadPool-5461-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2669139 INFO  (closeThreadPool-5461-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34467/solr
   [junit4]   2> 2669175 INFO  (zkConnectionManagerCallback-5483-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2669231 INFO  (zkConnectionManagerCallback-5485-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2669234 INFO  (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2669249 INFO  (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_    ] o.a.s.c.ZkController Publish node=127.0.0.1:37001_ as DOWN
   [junit4]   2> 2669250 INFO  (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2669250 INFO  (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37001_
   [junit4]   2> 2669251 INFO  (zkCallback-5470-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2669251 INFO  (zkCallback-5441-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2669252 INFO  (zkCallback-5459-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2669252 INFO  (zkCallback-5448-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2669305 INFO  (zkCallback-5477-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2669305 INFO  (zkCallback-5484-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2669361 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-3-001 of type NRT
   [junit4]   2> 2669415 WARN  (closeThreadPool-5461-thread-3) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2669415 INFO  (closeThreadPool-5461-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2669415 INFO  (closeThreadPool-5461-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2669416 INFO  (closeThreadPool-5461-thread-3) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2669507 INFO  (closeThreadPool-5461-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2669507 INFO  (closeThreadPool-5461-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2669507 INFO  (closeThreadPool-5461-thread-3) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2669508 INFO  (closeThreadPool-5461-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@575803bb{/,null,AVAILABLE}
   [junit4]   2> 2669508 WARN  (closeThreadPool-5461-thread-3) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@4e1f4151[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2669559 INFO  (zkConnectionManagerCallback-5492-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2669560 INFO  (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2669561 INFO  (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34467/solr ready
   [junit4]   2> 2669575 INFO  (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2669580 INFO  (closeThreadPool-5461-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5626e959{SSL,[ssl, http/1.1]}{127.0.0.1:40355}
   [junit4]   2> 2669580 INFO  (closeThreadPool-5461-thread-3) [    ] o.e.j.s.Server Started @2669664ms
   [junit4]   2> 2669580 INFO  (closeThreadPool-5461-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/tempDir-001/jetty3, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=40355, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-3-001/cores}
   [junit4]   2> 2669580 ERROR (closeThreadPool-5461-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2669580 INFO  (closeThreadPool-5461-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2669580 INFO  (closeThreadPool-5461-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.2
   [junit4]   2> 2669580 INFO  (closeThreadPool-5461-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2669580 INFO  (closeThreadPool-5461-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2669580 INFO  (closeThreadPool-5461-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-05-27T13:04:56.228Z
   [junit4]   2> 2669620 INFO  (zkConnectionManagerCallback-5495-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2669653 INFO  (closeThreadPool-5461-thread-3) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2669653 INFO  (closeThreadPool-5461-thread-3) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-3-001/solr.xml
   [junit4]   2> 2669656 INFO  (closeThreadPool-5461-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2669656 INFO  (closeThreadPool-5461-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2669670 INFO  (closeThreadPool-5461-thread-3) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2669714 INFO  (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
   [junit4]   2> 2669824 INFO  (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
   [junit4]   2> 2669824 INFO  (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
   [junit4]   2> 2669825 INFO  (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-2-001/cores
   [junit4]   2> 2670746 INFO  (closeThreadPool-5461-thread-3) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2670780 INFO  (closeThreadPool-5461-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34467/solr
   [junit4]   2> 2670855 INFO  (zkConnectionManagerCallback-5499-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2670928 INFO  (zkConnectionManagerCallback-5501-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2670931 INFO  (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2670951 INFO  (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_    ] o.a.s.c.ZkController Publish node=127.0.0.1:40355_ as DOWN
   [junit4]   2> 2670952 INFO  (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2670952 INFO  (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40355_
   [junit4]   2> 2670966 INFO  (zkCallback-5441-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2670982 INFO  (zkCallback-5470-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2670985 INFO  (zkCallback-5459-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2670985 INFO  (zkCallback-5448-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2671002 INFO  (zkCallback-5477-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2671002 INFO  (zkCallback-5484-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2671019 INFO  (zkCallback-5491-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2671034 INFO  (zkCallback-5500-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2671277 INFO  (zkConnectionManagerCallback-5508-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2671294 INFO  (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2671295 INFO  (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34467/solr ready
   [junit4]   2> 2671346 INFO  (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2671422 INFO  (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
   [junit4]   2> 2671494 INFO  (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
   [junit4]   2> 2671494 INFO  (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
   [junit4]   2> 2671512 INFO  (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-3-001/cores
   [junit4]   2> 2672463 INFO  (qtp712521339-39093) [n:127.0.0.1:33145_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:40355_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2672549 INFO  (qtp712521339-39097) [n:127.0.0.1:33145_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:33145_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2672696 INFO  (qtp712521339-39094) [n:127.0.0.1:33145_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:37001_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2672697 INFO  (OverseerThreadFactory-4550-thread-3-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_ c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:40355_ for creating new replica of shard shard2 for collection collection1
   [junit4]   2> 2672698 INFO  (OverseerThreadFactory-4550-thread-3-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_ c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 2672732 INFO  (OverseerThreadFactory-4550-thread-4-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:37001_ for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 2672733 INFO  (OverseerThreadFactory-4550-thread-4-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 2673079 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_    x:collection1_shard2_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n1&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2673114 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_    x:collection1_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n2&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2674271 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.2
   [junit4]   2> 2674306 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.2
   [junit4]   2> 2674574 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema [collection1_shard2_replica_n1] Schema name=test
   [junit4]   2> 2674577 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.s.IndexSchema [collection1_shard1_replica_n2] Schema name=test
   [junit4]   2> 2675251 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 2675399 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 2675456 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n2' using configuration from collection collection1, trusted=true
   [junit4]   2> 2675457 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n2' (registry 'solr.core.collection1.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
   [junit4]   2> 2675457 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SolrCore [[collection1_shard1_replica_n2] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-2-001/cores/collection1_shard1_replica_n2], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-2-001/cores/collection1_shard1_replica_n2/data/]
   [junit4]   2> 2675473 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=7, maxMergedSegmentMB=58.0263671875, floorSegmentMB=1.8046875, forceMergeDeletesPctAllowed=25.489696409831605, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=33.81642606487709
   [junit4]   2> 2675506 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 2675507 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n1' (registry 'solr.core.collection1.shard2.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
   [junit4]   2> 2675519 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [[collection1_shard2_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-3-001/cores/collection1_shard2_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-3-001/cores/collection1_shard2_replica_n1/data/]
   [junit4]   2> 2675522 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=7, maxMergedSegmentMB=58.0263671875, floorSegmentMB=1.8046875, forceMergeDeletesPctAllowed=25.489696409831605, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=33.81642606487709
   [junit4]   2> 2675642 WARN  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2675715 WARN  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2676021 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2676021 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2676055 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
   [junit4]   2> 2676055 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2676056 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@7e390287
   [junit4]   2> 2676078 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2676078 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2676080 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
   [junit4]   2> 2676080 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2676081 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@cc1eb8e
   [junit4]   2> 2676095 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1d5d8aaf[collection1_shard2_replica_n1] main]
   [junit4]   2> 2676095 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@cbb0764[collection1_shard1_replica_n2] main]
   [junit4]   2> 2676096 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2676096 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2676096 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2676096 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2676097 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2676097 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2676097 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1634690455563141120
   [junit4]   2> 2676097 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1634690455563141120
   [junit4]   2> 2676112 INFO  (searcherExecutor-4588-thread-1-processing-n:127.0.0.1:40355_ x:collection1_shard2_replica_n1 c:collection1 s:shard2) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [collection1_shard2_replica_n1] Registered new searcher Searcher@1d5d8aaf[collection1_shard2_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2676114 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node4=0}, version=0}
   [junit4]   2> 2676114 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard2
   [junit4]   2> 2676133 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2676133 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2676133 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:40355/collection1_shard2_replica_n1/
   [junit4]   2> 2676133 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2676133 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:40355/collection1_shard2_replica_n1/ has no replicas
   [junit4]   2> 2676133 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2676134 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard2/leader after winning as /collections/collection1/leader_elect/shard2/election/74764209601642515-core_node4-n_0000000000
   [junit4]   2> 2676134 INFO  (searcherExecutor-4587-thread-1-processing-n:127.0.0.1:37001_ x:collection1_shard1_replica_n2 c:collection1 s:shard1) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SolrCore [collection1_shard1_replica_n2] Registered new searcher Searcher@cbb0764[collection1_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2676148 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:40355/collection1_shard2_replica_n1/ shard2
   [junit4]   2> 2676150 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node3=0}, version=0}
   [junit4]   2> 2676150 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard1
   [junit4]   2> 2676169 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2676169 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2676169 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:37001/collection1_shard1_replica_n2/
   [junit4]   2> 2676169 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2676169 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SyncStrategy https://127.0.0.1:37001/collection1_shard1_replica_n2/ has no replicas
   [junit4]   2> 2676169 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2676169 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/74764209601642511-core_node3-n_0000000000
   [junit4]   2> 2676170 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:37001/collection1_shard1_replica_n2/ shard1
   [junit4]   2> 2676177 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2676305 INFO  (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n1&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3226
   [junit4]   2> 2676313 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2676476 INFO  (qtp712521339-39093) [n:127.0.0.1:33145_ c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:40355_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=4012
   [junit4]   2> 2676477 INFO  (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n2&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3363
   [junit4]   2> 2676565 WARN  (OverseerCollectionConfigSetProcessor-74764209601642501-127.0.0.1:35182_-n_0000000000) [n:127.0.0.1:35182_    ] o.a.s.c.LockTree lock_is_leaked at[collection1, shard1]
   [junit4]   2> 2676576 INFO  (OverseerThreadFactory-4550-thread-4-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_ c:collection1 s:shard1  ] o.a.s.c.LockTree Unlocked multiple times : collection1/shard1
   [junit4]   2> 2676585 INFO  (qtp712521339-39094) [n:127.0.0.1:33145_ c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:37001_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=3889
   [junit4]   2> 2676601 INFO  (OverseerThreadFactory-4550-thread-5-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_ c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:33145_ for creating new replica of shard shard2 for collection collection1
   [junit4]   2> 2676617 INFO  (OverseerCollectionConfigSetProcessor-74764209601642501-127.0.0.1:35182_-n_0000000000) [n:127.0.0.1:35182_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000008 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2676653 INFO  (OverseerCollectionConfigSetProcessor-74764209601642501-127.0.0.1:35182_-n_0000000000) [n:127.0.0.1:35182_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2676690 INFO  (OverseerThreadFactory-4550-thread-5-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_ c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 2676734 INFO  (qtp712521339-39093) [n:127.0.0.1:33145_    x:collection1_shard2_replica_n5] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n5&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2677852 INFO  (qtp712521339-39093) [n:127.0.0.1:33145_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.2
   [junit4]   2> 2677980 INFO  (qtp712521339-39093) [n:127.0.0.1:33145_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.s.IndexSchema [collection1_shard2_replica_n5] Schema name=test
   [junit4]   2> 2678326 INFO  (qtp712521339-39093) [n:127.0.0.1:33145_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 2678382 INFO  (qtp712521339-39093) [n:127.0.0.1:33145_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n5' using configuration from collection collection1, trusted=true
   [junit4]   2> 2678382 INFO  (qtp712521339-39093) [n:127.0.0.1:33145_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n5' (registry 'solr.core.collection1.shard2.replica_n5') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
   [junit4]   2> 2678383 INFO  (qtp712521339-39093) [n:127.0.0.1:33145_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.c.SolrCore [[collection1_shard2_replica_n5] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-1-001/cores/collection1_shard2_replica_n5], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-1-001/cores/collection1_shard2_replica_n5/data/]
   [junit4]   2> 2678398 INFO  (qtp712521339-39093) [n:127.0.0.1:33145_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=7, maxMergedSegmentMB=58.0263671875, floorSegmentMB=1.8046875, forceMergeDeletesPctAllowed=25.489696409831605, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=33.81642606487709
   [junit4]   2> 2678501 WARN  (qtp712521339-39093) [n:12

[...truncated too long message...]

cene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-authn-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-authz-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-changelog-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/api-all-1.0.0-M20.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/bcprov-jdk15on-1.60.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/byte-buddy-1.9.3.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/ehcache-core-2.4.4.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/hadoop-common-2.7.4-tests.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/hadoop-hdfs-2.7.4-tests.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/hadoop-minikdc-2.7.4.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/jersey-core-1.9.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/jersey-server-1.9.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/jetty-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/jetty-sslengine-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/jetty-util-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/mina-core-2.0.0-M5.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/mockito-core-2.23.4.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/netty-all-4.0.36.Final.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/objenesis-2.6.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/analysis/icu/lucene-analyzers-icu-7.7.2-SNAPSHOT.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/contrib/solr-analysis-extras/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/queryparser/classes/test:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/backward-codecs/classes/test:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/contrib/analysis-extras/lib/icu4j-62.1.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/usr/local/asfpackages/java/jdk1.8.0_191/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.7.2.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/temp/junit4-J0-20190527_122026_3735801384452750691395.events @/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/temp/junit4-J0-20190527_122026_3736170435525204442908.suites -stdin
   [junit4] ERROR: JVM J0 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1563)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:123)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:997)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:994)
   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4] at java.lang.Thread.run(Thread.java:748)

BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/build.xml:651: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/build.xml:577: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/build.xml:59: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build.xml:231: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/common-build.xml:558: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/common-build.xml:1571: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/common-build.xml:1098: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 450 minutes 37 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[Fast Archiver] No artifacts from Lucene-Solr-NightlyTests-7.7 #19 to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]