[JENKINS] Lucene-Solr-BadApples-NightlyTests-master - Build # 27 - Failure

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-NightlyTests-master - Build # 27 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-master/27/

4 tests failed.
FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv

Error Message:
Some docs had errors -- check logs expected:<0> but was:<2>

Stack Trace:
java.lang.AssertionError: Some docs had errors -- check logs expected:<0> but was:<2>
        at __randomizedtesting.SeedInfo.seed([116816C64F2C322:3702E32AEEAFF933]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:343)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv(TestStressCloudBlindAtomicUpdates.java:198)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv_stored_idx

Error Message:
expected:<471> but was:<376>

Stack Trace:
java.lang.AssertionError: expected:<471> but was:<376>
        at __randomizedtesting.SeedInfo.seed([116816C64F2C322:B31D04C1B846075]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:273)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv_stored_idx(TestStressCloudBlindAtomicUpdates.java:221)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
Could not find collection:delete_data_dir

Stack Trace:
java.lang.AssertionError: Could not find collection:delete_data_dir
        at __randomizedtesting.SeedInfo.seed([116816C64F2C322:8942BEB6CA0EAEDA]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:155)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:140)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:135)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:915)
        at org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:163)
        at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1008)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:983)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectField.<init>(DirectPostingsFormat.java:460)
        at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectFields.<init>(DirectPostingsFormat.java:131)
        at org.apache.lucene.codecs.memory.DirectPostingsFormat.fieldsProducer(DirectPostingsFormat.java:115)
        at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:293)
        at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:373)
        at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113)
        at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83)
        at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172)
        at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:214)
        at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106)
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:524)
        at org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:398)
        at org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:332)
        at org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit(TestDocTermOrdsUninvertLimit.java:67)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)




Build Log:
[...truncated 12942 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.StressHdfsTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/init-core-data-001
   [junit4]   2> 131438 WARN  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=18 numCloses=18
   [junit4]   2> 131438 INFO  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 131439 INFO  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 131439 INFO  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 131440 INFO  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /m_zsc/at
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 131471 WARN  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 131490 WARN  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 131495 INFO  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 131508 INFO  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_32848_hdfs____.a0qh5l/webapp
   [junit4]   2> 131889 INFO  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:32848
   [junit4]   2> 131993 WARN  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 131994 INFO  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 132017 INFO  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_46234_datanode____.w5c8co/webapp
   [junit4]   2> 132430 INFO  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46234
   [junit4]   2> 132458 WARN  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 132459 INFO  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 132474 INFO  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_40651_datanode____.pzirwj/webapp
   [junit4]   2> 132514 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to localhost/127.0.0.1:45034) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 132525 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xcd5174675f746: from storage DS-eeb507c6-5f96-4716-a80d-51d6d3021eba node DatanodeRegistration(127.0.0.1:35473, datanodeUuid=5f20a468-fe30-4a62-a89b-07c067f18dd0, infoPort=43863, infoSecurePort=0, ipcPort=33902, storageInfo=lv=-56;cid=testClusterID;nsid=538390404;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 132526 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xcd5174675f746: from storage DS-abbd6937-2cea-47ba-9959-95ab296323b8 node DatanodeRegistration(127.0.0.1:35473, datanodeUuid=5f20a468-fe30-4a62-a89b-07c067f18dd0, infoPort=43863, infoSecurePort=0, ipcPort=33902, storageInfo=lv=-56;cid=testClusterID;nsid=538390404;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 132867 INFO  (SUITE-StressHdfsTest-seed#[116816C64F2C322]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40651
   [junit4]   2> 132967 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to localhost/127.0.0.1:45034) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 132975 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xcd517614b6f9d: from storage DS-893b07e8-80fc-4b83-9db6-18254c9abe53 node DatanodeRegistration(127.0.0.1:46692, datanodeUuid=1ea59ab8-33b8-44e7-99e8-171d71dfc9b2, infoPort=34109, infoSecurePort=0, ipcPort=40802, storageInfo=lv=-56;cid=testClusterID;nsid=538390404;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 132975 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xcd517614b6f9d: from storage DS-852e1163-083f-4bf4-a4bd-511f1c8810b3 node DatanodeRegistration(127.0.0.1:46692, datanodeUuid=1ea59ab8-33b8-44e7-99e8-171d71dfc9b2, infoPort=34109, infoSecurePort=0, ipcPort=40802, storageInfo=lv=-56;cid=testClusterID;nsid=538390404;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 133048 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 133048 INFO  (Thread-3531) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 133048 INFO  (Thread-3531) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 133051 ERROR (Thread-3531) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 133149 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.ZkTestServer start zk server on port:42365
   [junit4]   2> 133154 INFO  (zkConnectionManagerCallback-158-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 133165 INFO  (zkConnectionManagerCallback-160-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 133170 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 133172 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 133173 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 133174 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 133176 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 133177 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 133178 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 133179 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 133180 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 133181 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 133182 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 133185 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use TLOG replicas unless explicitly asked otherwise
   [junit4]   2> 133292 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 133294 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 133294 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 133294 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 133295 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@21f92e0b{/m_zsc/at,null,AVAILABLE}
   [junit4]   2> 133296 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3560c7f6{HTTP/1.1,[http/1.1]}{127.0.0.1:42982}
   [junit4]   2> 133296 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.Server Started @133334ms
   [junit4]   2> 133296 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:45034/hdfs__localhost_45034__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-BadApples-NightlyTests-master_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001_tempDir-002_control_data, hostContext=/m_zsc/at, hostPort=42982, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/control-001/cores}
   [junit4]   2> 133297 ERROR (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 133297 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 133297 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 133297 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 133297 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 133297 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-07T12:27:18.364Z
   [junit4]   2> 133299 INFO  (zkConnectionManagerCallback-162-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 133300 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 133300 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/control-001/solr.xml
   [junit4]   2> 133304 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 133304 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 133305 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@77855ce8, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 133378 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42365/solr
   [junit4]   2> 133384 INFO  (zkConnectionManagerCallback-166-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 133412 INFO  (zkConnectionManagerCallback-168-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 133503 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42982_m_zsc%2Fat
   [junit4]   2> 133504 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.c.Overseer Overseer (id=72294309820170244-127.0.0.1:42982_m_zsc%2Fat-n_0000000000) starting
   [junit4]   2> 133512 INFO  (zkConnectionManagerCallback-175-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 133514 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42365/solr ready
   [junit4]   2> 133515 INFO  (OverseerStateUpdate-72294309820170244-127.0.0.1:42982_m_zsc%2Fat-n_0000000000) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:42982_m_zsc%2Fat
   [junit4]   2> 133516 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42982_m_zsc%2Fat
   [junit4]   2> 133526 INFO  (zkCallback-174-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 133536 INFO  (zkCallback-167-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 133538 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 133553 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77855ce8
   [junit4]   2> 133559 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77855ce8
   [junit4]   2> 133559 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77855ce8
   [junit4]   2> 133561 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/control-001/cores
   [junit4]   2> 133581 INFO  (zkConnectionManagerCallback-181-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 133582 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 133583 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42365/solr ready
   [junit4]   2> 133585 INFO  (qtp130022785-4382) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:42982_m_zsc%252Fat&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 133587 INFO  (OverseerThreadFactory-594-thread-1-processing-n:127.0.0.1:42982_m_zsc%2Fat) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 133695 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 133696 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 134713 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 134725 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 134853 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 134868 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 134868 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77855ce8
   [junit4]   2> 134869 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:45034/solr_hdfs_home
   [junit4]   2> 134869 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 134869 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[hdfs://localhost:45034/solr_hdfs_home/control_collection/core_node2/data/]
   [junit4]   2> 134870 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:45034/solr_hdfs_home/control_collection/core_node2/data/snapshot_metadata
   [junit4]   2> 134878 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 134878 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 134878 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 134897 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 134898 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:45034/solr_hdfs_home/control_collection/core_node2/data
   [junit4]   2> 134927 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:45034/solr_hdfs_home/control_collection/core_node2/data/index
   [junit4]   2> 134933 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 134933 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 134933 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 134940 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 134941 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@5604e3f5
   [junit4]   2> 134970 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35473 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-852e1163-083f-4bf4-a4bd-511f1c8810b3:NORMAL:127.0.0.1:46692|RBW], ReplicaUC[[DISK]DS-abbd6937-2cea-47ba-9959-95ab296323b8:NORMAL:127.0.0.1:35473|RBW]]} size 0
   [junit4]   2> 134971 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46692 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-852e1163-083f-4bf4-a4bd-511f1c8810b3:NORMAL:127.0.0.1:46692|RBW], ReplicaUC[[DISK]DS-abbd6937-2cea-47ba-9959-95ab296323b8:NORMAL:127.0.0.1:35473|RBW]]} size 0
   [junit4]   2> 134976 WARN  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 135042 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 135042 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 135042 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 135053 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 135053 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 135054 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=8, maxMergeAtOnceExplicit=5, maxMergedSegmentMB=59.814453125, floorSegmentMB=1.80859375, forceMergeDeletesPctAllowed=25.471104967824605, segmentsPerTier=13.0, maxCFSSegmentSizeMB=1.23828125, noCFSRatio=0.1938771048452849, deletesPctAllowed=43.06110451695383
   [junit4]   2> 135100 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6db076b9[control_collection_shard1_replica_n1] main]
   [junit4]   2> 135103 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 135103 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 135104 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 135112 INFO  (searcherExecutor-599-thread-1-processing-n:127.0.0.1:42982_m_zsc%2Fat x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@6db076b9[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 135112 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1610951677893935104
   [junit4]   2> 135130 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 135138 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 135138 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 135138 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:42982/m_zsc/at/control_collection_shard1_replica_n1/
   [junit4]   2> 135139 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 135139 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:42982/m_zsc/at/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 135139 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 135144 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:42982/m_zsc/at/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 135246 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 135252 INFO  (qtp130022785-4377) [n:127.0.0.1:42982_m_zsc%2Fat c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1557
   [junit4]   2> 135256 INFO  (qtp130022785-4382) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 135352 INFO  (zkCallback-167-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 135589 INFO  (OverseerCollectionConfigSetProcessor-72294309820170244-127.0.0.1:42982_m_zsc%2Fat-n_0000000000) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 136256 INFO  (qtp130022785-4382) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:42982_m_zsc%252Fat&wt=javabin&version=2} status=0 QTime=2671
   [junit4]   2> 136262 INFO  (zkConnectionManagerCallback-186-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 136263 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 136264 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42365/solr ready
   [junit4]   2> 136267 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 136269 INFO  (qtp130022785-4378) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 136271 INFO  (OverseerThreadFactory-594-thread-2-processing-n:127.0.0.1:42982_m_zsc%2Fat) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 136475 WARN  (OverseerThreadFactory-594-thread-2-processing-n:127.0.0.1:42982_m_zsc%2Fat) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 136477 INFO  (qtp130022785-4378) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 136477 INFO  (qtp130022785-4378) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=208
   [junit4]   2> 136580 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/shard-1-001 of type TLOG
   [junit4]   2> 136581 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 136582 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 136582 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 136582 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 136582 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5b60a2ca{/m_zsc/at,null,AVAILABLE}
   [junit4]   2> 136588 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@11d71063{HTTP/1.1,[http/1.1]}{127.0.0.1:42578}
   [junit4]   2> 136588 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.Server Started @136627ms
   [junit4]   2> 136589 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:45034/hdfs__localhost_45034__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-BadApples-NightlyTests-master_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001_tempDir-002_jetty1, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/m_zsc/at, hostPort=42578, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/shard-1-001/cores}
   [junit4]   2> 136589 ERROR (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 136589 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 136589 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 136589 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 136589 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 136589 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-07T12:27:21.656Z
   [junit4]   2> 136591 INFO  (zkConnectionManagerCallback-188-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 136593 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 136593 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/shard-1-001/solr.xml
   [junit4]   2> 136597 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 136597 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 136599 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@77855ce8, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 136912 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42365/solr
   [junit4]   2> 136914 INFO  (zkConnectionManagerCallback-192-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 136923 INFO  (zkConnectionManagerCallback-194-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 136935 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 136938 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 136938 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42578_m_zsc%2Fat
   [junit4]   2> 136940 INFO  (zkCallback-167-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 136940 INFO  (zkCallback-185-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 136940 INFO  (zkCallback-174-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 136941 INFO  (zkCallback-193-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 136964 INFO  (zkConnectionManagerCallback-201-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 136965 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 136966 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42365/solr ready
   [junit4]   2> 136966 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 136983 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77855ce8
   [junit4]   2> 136991 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77855ce8
   [junit4]   2> 136991 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77855ce8
   [junit4]   2> 136992 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/shard-1-001/cores
   [junit4]   2> 137024 INFO  (qtp190173436-4449) [n:127.0.0.1:42578_m_zsc%2Fat    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:42578_m_zsc%252Fat&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 137027 INFO  (OverseerCollectionConfigSetProcessor-72294309820170244-127.0.0.1:42982_m_zsc%2Fat-n_0000000000) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 137029 INFO  (OverseerThreadFactory-594-thread-3-processing-n:127.0.0.1:42982_m_zsc%2Fat) [n:127.0.0.1:42982_m_zsc%2Fat c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:42578_m_zsc%2Fat for creating new replica of shard shard1
   [junit4]   2> 137034 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat    x:collection1_shard1_replica_t21] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 138058 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 138070 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.IndexSchema [collection1_shard1_replica_t21] Schema name=test
   [junit4]   2> 138180 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 138211 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t21' using configuration from collection collection1, trusted=true
   [junit4]   2> 138211 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_t21' (registry 'solr.core.collection1.shard1.replica_t21') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77855ce8
   [junit4]   2> 138211 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:45034/solr_hdfs_home
   [junit4]   2> 138211 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 138212 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SolrCore [[collection1_shard1_replica_t21] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/shard-1-001/cores/collection1_shard1_replica_t21], dataDir=[hdfs://localhost:45034/solr_hdfs_home/collection1/core_node22/data/]
   [junit4]   2> 138213 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:45034/solr_hdfs_home/collection1/core_node22/data/snapshot_metadata
   [junit4]   2> 138221 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 138221 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 138221 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 138230 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 138231 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:45034/solr_hdfs_home/collection1/core_node22/data
   [junit4]   2> 138300 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:45034/solr_hdfs_home/collection1/core_node22/data/index
   [junit4]   2> 138326 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 138326 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 138326 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 138386 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 138390 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@78b42455
   [junit4]   2> 138453 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46692 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-eeb507c6-5f96-4716-a80d-51d6d3021eba:NORMAL:127.0.0.1:35473|RBW], ReplicaUC[[DISK]DS-893b07e8-80fc-4b83-9db6-18254c9abe53:NORMAL:127.0.0.1:46692|FINALIZED]]} size 0
   [junit4]   2> 138457 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35473 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-eeb507c6-5f96-4716-a80d-51d6d3021eba:NORMAL:127.0.0.1:35473|RBW], ReplicaUC[[DISK]DS-893b07e8-80fc-4b83-9db6-18254c9abe53:NORMAL:127.0.0.1:46692|FINALIZED]]} size 0
   [junit4]   2> 138463 WARN  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 138541 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 138541 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 138541 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 138552 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 138553 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 138554 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=8, maxMergeAtOnceExplicit=5, maxMergedSegmentMB=59.814453125, floorSegmentMB=1.80859375, forceMergeDeletesPctAllowed=25.471104967824605, segmentsPerTier=13.0, maxCFSSegmentSizeMB=1.23828125, noCFSRatio=0.1938771048452849, deletesPctAllowed=43.06110451695383
   [junit4]   2> 138567 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.SolrIndexSearcher Opening [Searcher@604a05d8[collection1_shard1_replica_t21] main]
   [junit4]   2> 138569 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 138569 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 138570 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 138571 INFO  (searcherExecutor-613-thread-1-processing-n:127.0.0.1:42578_m_zsc%2Fat x:collection1_shard1_replica_t21 c:collection1 s:shard1) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SolrCore [collection1_shard1_replica_t21] Registered new searcher Searcher@604a05d8[collection1_shard1_replica_t21] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 138571 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1610951681520959488
   [junit4]   2> 138579 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node22=0}, version=0}
   [junit4]   2> 138582 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 138582 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 138582 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:42578/m_zsc/at/collection1_shard1_replica_t21/
   [junit4]   2> 138582 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 138582 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SyncStrategy http://127.0.0.1:42578/m_zsc/at/collection1_shard1_replica_t21/ has no replicas
   [junit4]   2> 138582 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 138582 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ZkController collection1_shard1_replica_t21 stopping background replication from leader
   [junit4]   2> 138585 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:42578/m_zsc/at/collection1_shard1_replica_t21/ shard1
   [junit4]   2> 138736 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 138739 INFO  (qtp190173436-4444) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1705
   [junit4]   2> 138745 INFO  (qtp190173436-4449) [n:127.0.0.1:42578_m_zsc%2Fat c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:42578_m_zsc%252Fat&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2} status=0 QTime=1721
   [junit4]   2> 138962 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/shard-2-001 of type TLOG
   [junit4]   2> 138963 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 138965 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 138965 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 138965 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 138965 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@d00273e{/m_zsc/at,null,AVAILABLE}
   [junit4]   2> 138968 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2efa5edc{HTTP/1.1,[http/1.1]}{127.0.0.1:34475}
   [junit4]   2> 138968 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.e.j.s.Server Started @139006ms
   [junit4]   2> 138968 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:45034/hdfs__localhost_45034__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-BadApples-NightlyTests-master_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001_tempDir-002_jetty2, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/m_zsc/at, hostPort=34475, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/shard-2-001/cores}
   [junit4]   2> 138969 ERROR (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 138969 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 138969 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 138969 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 138969 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 138969 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-07T12:27:24.036Z
   [junit4]   2> 138981 INFO  (zkConnectionManagerCallback-204-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 138986 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 138986 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/shard-2-001/solr.xml
   [junit4]   2> 138990 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 138990 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 138991 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@77855ce8, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 139028 INFO  (OverseerCollectionConfigSetProcessor-72294309820170244-127.0.0.1:42982_m_zsc%2Fat-n_0000000000) [n:127.0.0.1:42982_m_zsc%2Fat    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 139225 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42365/solr
   [junit4]   2> 139227 INFO  (zkConnectionManagerCallback-208-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 139230 INFO  (zkConnectionManagerCallback-210-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 139237 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 139241 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 139241 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34475_m_zsc%2Fat
   [junit4]   2> 139243 INFO  (zkCallback-174-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 139243 INFO  (zkCallback-193-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 139243 INFO  (zkCallback-185-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 139243 INFO  (zkCallback-200-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 139243 INFO  (zkCallback-167-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 139244 INFO  (zkCallback-209-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 139263 INFO  (zkConnectionManagerCallback-217-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 139264 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 139265 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42365/solr ready
   [junit4]   2> 139265 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 139282 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77855ce8
   [junit4]   2> 139289 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77855ce8
   [junit4]   2> 139289 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77855ce8
   [junit4]   2> 139290 INFO  (TEST-StressHdfsTest.test-seed#[116816C64F2C322]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/shard-2-001/cores
   [junit4]   2> 139319 INFO  (qtp699522644-4495) [n:127.0.0.1:34475_m_zsc%2Fat    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:34475_m_zsc%252Fat&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 139323 INFO  (OverseerThreadFactory-594-thread-4-processing-n:127.0.0.1:42982_m_zsc%2Fat) [n:127.0.0.1:42982_m_zsc%2Fat c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:34475_m_zsc%2Fat for creating new replica of shard shard1
   [junit4]   2> 139336 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat    x:collection1_shard1_replica_t23] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t23&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 140349 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 140362 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.s.IndexSchema [collection1_shard1_replica_t23] Schema name=test
   [junit4]   2> 140449 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 140462 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t23' using configuration from collection collection1, trusted=true
   [junit4]   2> 140462 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_t23' (registry 'solr.core.collection1.shard1.replica_t23') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77855ce8
   [junit4]   2> 140462 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:45034/solr_hdfs_home
   [junit4]   2> 140462 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 140462 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.c.SolrCore [[collection1_shard1_replica_t23] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_116816C64F2C322-001/shard-2-001/cores/collection1_shard1_replica_t23], dataDir=[hdfs://localhost:45034/solr_hdfs_home/collection1/core_node24/data/]
   [junit4]   2> 140463 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:45034/solr_hdfs_home/collection1/core_node24/data/snapshot_metadata
   [junit4]   2> 140470 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 140470 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 140470 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 140482 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 140483 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:45034/solr_hdfs_home/collection1/core_node24/data
   [junit4]   2> 140502 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:45034/solr_hdfs_home/collection1/core_node24/data/index
   [junit4]   2> 140508 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 140508 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 140508 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 140514 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 140515 INFO  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@66833cc3
   [junit4]   2> 140532 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35473 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-893b07e8-80fc-4b83-9db6-18254c9abe53:NORMAL:127.0.0.1:46692|RBW], ReplicaUC[[DISK]DS-abbd6937-2cea-47ba-9959-95ab296323b8:NORMAL:127.0.0.1:35473|FINALIZED]]} size 0
   [junit4]   2> 140533 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46692 is added to blk_1073741827_1003 size 69
   [junit4]   2> 140536 WARN  (qtp699522644-4490) [n:127.0.0.1:34475_m_zsc%2Fat c:collection1 s:shard1  x:collection1_shard1_replica_t23] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 140580 INFO  (qtp699522

[...truncated too long message...]

}
   [junit4]   2> 2871051 INFO  (jetty-closer-5889-thread-7) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1672510e{/solr,null,UNAVAILABLE}
   [junit4]   2> 2871051 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100d77094860019, likely client has closed socket
   [junit4]   2> 2871051 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100d7709486001b, likely client has closed socket
   [junit4]   2> 2871051 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100d77094860023, likely client has closed socket
   [junit4]   2> 2871051 INFO  (jetty-closer-5889-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@31540f4e{/solr,null,UNAVAILABLE}
   [junit4]   2> 2871051 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100d7709486001c, likely client has closed socket
   [junit4]   2> 2871051 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100d7709486001a, likely client has closed socket
   [junit4]   2> 2871059 INFO  (jetty-closer-5889-thread-4) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 2871059 INFO  (jetty-closer-5889-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@580bb75c{/solr,null,UNAVAILABLE}
   [junit4]   2> 2871059 INFO  (jetty-closer-5889-thread-1) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 2871059 INFO  (jetty-closer-5889-thread-5) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 2871059 INFO  (jetty-closer-5889-thread-7) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 2871059 INFO  (jetty-closer-5889-thread-3) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 2872548 INFO  (jetty-closer-5889-thread-9) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7a7f5821{/solr,null,UNAVAILABLE}
   [junit4]   2> 2872548 INFO  (jetty-closer-5889-thread-9) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 2872550 INFO  (jetty-closer-5889-thread-8) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@66718dd6{/solr,null,UNAVAILABLE}
   [junit4]   2> 2872550 INFO  (jetty-closer-5889-thread-8) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 2872551 INFO  (jetty-closer-5889-thread-10) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7e719202{/solr,null,UNAVAILABLE}
   [junit4]   2> 2872551 INFO  (jetty-closer-5889-thread-10) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 2872552 INFO  (jetty-closer-5889-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@15ccbdb1{/solr,null,UNAVAILABLE}
   [junit4]   2> 2872552 INFO  (jetty-closer-5889-thread-2) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 2872553 ERROR (SUITE-TestStressCloudBlindAtomicUpdates-seed#[116816C64F2C322]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2872553 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[116816C64F2C322]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41808 41808
   [junit4]   2> 2872649 INFO  (Thread-12297) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41808 41808
   [junit4]   2> 2872651 WARN  (Thread-12297) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 119 /solr/collections/test_col/terms/shard1
   [junit4]   2> 105 /solr/collections/test_col/terms/shard4
   [junit4]   2> 104 /solr/collections/test_col/terms/shard5
   [junit4]   2> 101 /solr/collections/test_col/terms/shard2
   [junit4]   2> 99 /solr/collections/test_col/terms/shard3
   [junit4]   2> 21 /solr/aliases.json
   [junit4]   2> 21 /solr/clusterprops.json
   [junit4]   2> 10 /solr/configs/org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set
   [junit4]   2> 10 /solr/security.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 21 /solr/clusterstate.json
   [junit4]   2> 10 /solr/collections/test_col/state.json
   [junit4]   2> 2 /solr/overseer_elect/election/72294472566046742-127.0.0.1:35841_solr-n_0000000007
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 21 /solr/live_nodes
   [junit4]   2> 21 /solr/collections
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_116816C64F2C322-001
   [junit4]   2> Sep 07, 2018 1:12:57 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@4df6f933), locale=hi-IN, timezone=Europe/Busingen
   [junit4]   2> NOTE: Linux 4.4.0-130-generic amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=4,threads=1,free=195793328,total=528482304
   [junit4]   2> NOTE: All tests run in this JVM: [AtomicUpdateProcessorFactoryTest, HdfsDirectoryTest, JvmMetricsTest, TestStressRecovery, TestInPlaceUpdatesStandalone, TestSimNodeAddedTrigger, TestRealTimeGet, BasicAuthIntegrationTest, TestSolrCloudWithDelegationTokens, DistributedSpellCheckComponentTest, HdfsUnloadDistributedZkTest, SimplePostToolTest, TestJsonFacetRefinement, DistributedQueryComponentCustomSortTest, DateMathParserTest, TestStreamBody, CollectionPropsTest, ImplicitSnitchTest, QueryParsingTest, TestPerFieldSimilarity, TestFuzzyAnalyzedSuggestions, TestOmitPositions, LeaderFailoverAfterPartitionTest, TestMaxTokenLenTokenizer, SmileWriterTest, SuggesterWFSTTest, TestSchemaResource, ConcurrentDeleteAndCreateCollectionTest, TestSerializedLuceneMatchVersion, DistributedFacetPivotSmallTest, DeleteShardTest, StatelessScriptUpdateProcessorFactoryTest, TriggerIntegrationTest, IndexSchemaRuntimeFieldTest, TestLFUCache, PreAnalyzedFieldTest, TermVectorComponentDistributedTest, TestDFISimilarityFactory, TestShortCircuitedRequests, NodeLostTriggerTest, ChaosMonkeySafeLeaderWithPullReplicasTest, SpellCheckCollatorTest, TestCustomDocTransformer, TestDocTermOrdsUninvertLimit, AutoAddReplicasPlanActionTest, Tagger2Test, TestCloudConsistency, TestLMDirichletSimilarityFactory, LukeRequestHandlerTest, TestReplicaProperties, CSVRequestHandlerTest, FullSolrCloudDistribCmdsTest, DefaultValueUpdateProcessorTest, TestBulkSchemaAPI, SolrXmlInZkTest, XmlInterpolationTest, TestMaxScoreQueryParser, ReplaceNodeTest, PathHierarchyTokenizerFactoryTest, OverseerRolesTest, LeaderVoteWaitTimeoutTest, DateRangeFieldTest, TestStressVersions, TestDeleteCollectionOnDownNodes, TestUseDocValuesAsStored2, SchemaApiFailureTest, JsonLoaderTest, SolrGangliaReporterTest, TestCloudSearcherWarming, SearchHandlerTest, HDFSCollectionsAPITest, ZkFailoverTest, HdfsSyncSliceTest, TestPayloadScoreQParserPlugin, DistribDocExpirationUpdateProcessorTest, UpdateParamsTest, SpatialRPTFieldTypeTest, TestConfigSetProperties, DistributedVersionInfoTest, TestExceedMaxTermLength, RollingRestartTest, AnalysisErrorHandlingTest, AlternateDirectoryTest, FieldMutatingUpdateProcessorTest, SystemInfoHandlerTest, TestRangeQuery, DirectSolrSpellCheckerTest, TestConfig, SolrJmxReporterTest, TestManagedResource, ChaosMonkeyNothingIsSafeTest, TestRandomFlRTGCloud, LargeFieldTest, TestSolrIndexConfig, BasicDistributedZk2Test, MultiTermTest, CopyFieldTest, TestHashQParserPlugin, HdfsRecoverLeaseTest, TestBlobHandler, TestFieldCacheWithThreads, TestBackupRepositoryFactory, TestJsonFacetsWithNestedObjects, CursorPagingTest, TestDynamicFieldResource, TestSimTriggerIntegration, TestScoreJoinQPScore, BadIndexSchemaTest, BlockJoinFacetRandomTest, TestSubQueryTransformer, ParsingFieldUpdateProcessorsTest, CollectionTooManyReplicasTest, ResponseHeaderTest, SimpleFacetsTest, TestConfigsApi, TestMinMaxOnMultiValuedField, TestGroupingSearch, TestSolrQueryResponse, TestStressCloudBlindAtomicUpdates]
   [junit4] Completed [348/834 (3!)] on J0 in 256.58s, 5 tests, 2 failures <<< FAILURES!

[...truncated 1538 lines...]
   [junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J0-20180907_122504_984654396407922203023.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/heapdumps/java_pid23574.hprof ...
   [junit4] Heap dump file created [473378627 bytes in 2.515 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 8828 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/build.xml:661: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/build.xml:585: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid23574.hprof

Total time: 293 minutes 19 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-NightlyTests-master - Build # 28 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-master/28/

4 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test

Error Message:
Error from server at <a href="http://127.0.0.1:45370:">http://127.0.0.1:45370: ADDREPLICA failed to create replica

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at <a href="http://127.0.0.1:45370:">http://127.0.0.1:45370: ADDREPLICA failed to create replica
        at __randomizedtesting.SeedInfo.seed([E3C53B70919B912C:6B9104AA3F67FCD4]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.UnloadDistributedZkTest.testUnloadShardAndCollection(UnloadDistributedZkTest.java:126)
        at org.apache.solr.cloud.UnloadDistributedZkTest.test(UnloadDistributedZkTest.java:73)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1008)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:983)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest.testPreemptiveCreation

Error Message:
Captured an uncaught exception in thread: Thread[id=4512, name=TRA-preemptive-creation-1889-thread-1, state=RUNNABLE, group=TGRP-TimeRoutedAliasUpdateProcessorTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4512, name=TRA-preemptive-creation-1889-thread-1, state=RUNNABLE, group=TGRP-TimeRoutedAliasUpdateProcessorTest]
Caused by: org.apache.solr.common.SolrException
        at __randomizedtesting.SeedInfo.seed([E3C53B70919B912C]:0)
        at org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)
        at org.apache.solr.cloud.api.collections.MaintainRoutedAliasCmd.remoteInvoke(MaintainRoutedAliasCmd.java:93)
        at org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessor.createNextCollection(TimeRoutedAliasUpdateProcessor.java:359)
        at org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessor.lambda$createCollectionsIfRequired$0(TimeRoutedAliasUpdateProcessor.java:228)
        at org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessor.lambda$preemptiveAsync$1(TimeRoutedAliasUpdateProcessor.java:253)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest:     1) Thread[id=4525, name=TRA-preemptive-creation-1889-thread-2, state=WAITING, group=TGRP-TimeRoutedAliasUpdateProcessorTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest:
   1) Thread[id=4525, name=TRA-preemptive-creation-1889-thread-2, state=WAITING, group=TGRP-TimeRoutedAliasUpdateProcessorTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([E3C53B70919B912C]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=4525, name=TRA-preemptive-creation-1889-thread-2, state=WAITING, group=TGRP-TimeRoutedAliasUpdateProcessorTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=4525, name=TRA-preemptive-creation-1889-thread-2, state=WAITING, group=TGRP-TimeRoutedAliasUpdateProcessorTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([E3C53B70919B912C]:0)




Build Log:
[...truncated 13656 lines...]
   [junit4] Suite: org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/init-core-data-001
   [junit4]   2> 654120 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[E3C53B70919B912C]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 654121 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[E3C53B70919B912C]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 654122 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[E3C53B70919B912C]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 654122 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[E3C53B70919B912C]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001
   [junit4]   2> 654122 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[E3C53B70919B912C]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 654127 INFO  (Thread-1004) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 654127 INFO  (Thread-1004) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 654129 ERROR (Thread-1004) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 654228 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[E3C53B70919B912C]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:44910
   [junit4]   2> 654229 INFO  (zkConnectionManagerCallback-1064-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 654255 INFO  (jetty-launcher-1061-thread-3) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 654256 INFO  (jetty-launcher-1061-thread-2) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 654257 INFO  (jetty-launcher-1061-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 654257 INFO  (jetty-launcher-1061-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 654257 INFO  (jetty-launcher-1061-thread-3) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 654257 INFO  (jetty-launcher-1061-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@68b96c41{/solr,null,AVAILABLE}
   [junit4]   2> 654258 INFO  (jetty-launcher-1061-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3ca13773{SSL,[ssl, http/1.1]}{127.0.0.1:45836}
   [junit4]   2> 654258 INFO  (jetty-launcher-1061-thread-3) [    ] o.e.j.s.Server Started @654321ms
   [junit4]   2> 654258 INFO  (jetty-launcher-1061-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=45836}
   [junit4]   2> 654258 ERROR (jetty-launcher-1061-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 654258 INFO  (jetty-launcher-1061-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 654258 INFO  (jetty-launcher-1061-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 654258 INFO  (jetty-launcher-1061-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 654258 INFO  (jetty-launcher-1061-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 654259 INFO  (jetty-launcher-1061-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-15T02:11:15.463Z
   [junit4]   2> 654259 INFO  (jetty-launcher-1061-thread-1) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 654260 INFO  (jetty-launcher-1061-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 654261 INFO  (jetty-launcher-1061-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 654261 INFO  (jetty-launcher-1061-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 654262 INFO  (jetty-launcher-1061-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1e34c624{/solr,null,AVAILABLE}
   [junit4]   2> 654262 INFO  (jetty-launcher-1061-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2902721c{SSL,[ssl, http/1.1]}{127.0.0.1:36242}
   [junit4]   2> 654262 INFO  (jetty-launcher-1061-thread-1) [    ] o.e.j.s.Server Started @654325ms
   [junit4]   2> 654262 INFO  (jetty-launcher-1061-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36242}
   [junit4]   2> 654262 INFO  (zkConnectionManagerCallback-1066-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 654262 ERROR (jetty-launcher-1061-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 654263 INFO  (jetty-launcher-1061-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 654263 INFO  (jetty-launcher-1061-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 654263 INFO  (jetty-launcher-1061-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 654263 INFO  (jetty-launcher-1061-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 654263 INFO  (jetty-launcher-1061-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-15T02:11:15.468Z
   [junit4]   2> 654264 INFO  (jetty-launcher-1061-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 654271 INFO  (zkConnectionManagerCallback-1068-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 654279 INFO  (jetty-launcher-1061-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 654279 INFO  (jetty-launcher-1061-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 654279 INFO  (jetty-launcher-1061-thread-2) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 654280 INFO  (jetty-launcher-1061-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@172d17d0{/solr,null,AVAILABLE}
   [junit4]   2> 654280 INFO  (jetty-launcher-1061-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7e1476c{SSL,[ssl, http/1.1]}{127.0.0.1:43613}
   [junit4]   2> 654280 INFO  (jetty-launcher-1061-thread-2) [    ] o.e.j.s.Server Started @654343ms
   [junit4]   2> 654280 INFO  (jetty-launcher-1061-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43613}
   [junit4]   2> 654280 ERROR (jetty-launcher-1061-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 654280 INFO  (jetty-launcher-1061-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 654281 INFO  (jetty-launcher-1061-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 654281 INFO  (jetty-launcher-1061-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 654281 INFO  (jetty-launcher-1061-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 654281 INFO  (jetty-launcher-1061-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-15T02:11:15.486Z
   [junit4]   2> 654286 INFO  (jetty-launcher-1061-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 654323 INFO  (jetty-launcher-1061-thread-4) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 654354 INFO  (zkConnectionManagerCallback-1070-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 654354 INFO  (jetty-launcher-1061-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 654401 INFO  (jetty-launcher-1061-thread-4) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 654401 INFO  (jetty-launcher-1061-thread-4) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 654401 INFO  (jetty-launcher-1061-thread-4) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 654410 INFO  (jetty-launcher-1061-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6511ef1d{/solr,null,AVAILABLE}
   [junit4]   2> 654411 INFO  (jetty-launcher-1061-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@17cfaa47{SSL,[ssl, http/1.1]}{127.0.0.1:44492}
   [junit4]   2> 654411 INFO  (jetty-launcher-1061-thread-4) [    ] o.e.j.s.Server Started @654474ms
   [junit4]   2> 654411 INFO  (jetty-launcher-1061-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44492}
   [junit4]   2> 654411 ERROR (jetty-launcher-1061-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 654411 INFO  (jetty-launcher-1061-thread-4) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 654411 INFO  (jetty-launcher-1061-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 654411 INFO  (jetty-launcher-1061-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 654412 INFO  (jetty-launcher-1061-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 654412 INFO  (jetty-launcher-1061-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-15T02:11:15.617Z
   [junit4]   2> 654446 INFO  (zkConnectionManagerCallback-1072-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 654446 INFO  (jetty-launcher-1061-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 654628 INFO  (jetty-launcher-1061-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44910/solr
   [junit4]   2> 654642 INFO  (jetty-launcher-1061-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44910/solr
   [junit4]   2> 654643 INFO  (jetty-launcher-1061-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44910/solr
   [junit4]   2> 654653 INFO  (zkConnectionManagerCallback-1077-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 654687 INFO  (zkConnectionManagerCallback-1081-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 654688 INFO  (zkConnectionManagerCallback-1084-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 654688 INFO  (zkConnectionManagerCallback-1086-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 654712 INFO  (jetty-launcher-1061-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44910/solr
   [junit4]   2> 654713 INFO  (zkConnectionManagerCallback-1090-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 654714 INFO  (zkConnectionManagerCallback-1094-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 654721 INFO  (zkConnectionManagerCallback-1088-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 654740 INFO  (zkConnectionManagerCallback-1096-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 654765 INFO  (jetty-launcher-1061-thread-4) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:44492_solr
   [junit4]   2> 654767 INFO  (jetty-launcher-1061-thread-4) [n:127.0.0.1:44492_solr    ] o.a.s.c.Overseer Overseer (id=73323413769158668-127.0.0.1:44492_solr-n_0000000000) starting
   [junit4]   2> 654769 INFO  (jetty-launcher-1061-thread-3) [n:127.0.0.1:45836_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45836_solr
   [junit4]   2> 654782 INFO  (zkCallback-1087-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 654788 INFO  (zkCallback-1095-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 654846 INFO  (zkConnectionManagerCallback-1114-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 654847 INFO  (zkConnectionManagerCallback-1110-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 654847 INFO  (jetty-launcher-1061-thread-4) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 654848 INFO  (jetty-launcher-1061-thread-3) [n:127.0.0.1:45836_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 654851 INFO  (jetty-launcher-1061-thread-4) [n:127.0.0.1:44492_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44910/solr ready
   [junit4]   2> 654851 INFO  (jetty-launcher-1061-thread-3) [n:127.0.0.1:45836_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44910/solr ready
   [junit4]   2> 654852 INFO  (OverseerStateUpdate-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:44492_solr
   [junit4]   2> 654852 INFO  (jetty-launcher-1061-thread-3) [n:127.0.0.1:45836_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 654858 INFO  (jetty-launcher-1061-thread-4) [n:127.0.0.1:44492_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 654858 INFO  (jetty-launcher-1061-thread-4) [n:127.0.0.1:44492_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44492_solr
   [junit4]   2> 654864 INFO  (zkCallback-1113-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 654866 INFO  (zkCallback-1087-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 654866 INFO  (zkCallback-1095-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 654874 INFO  (zkCallback-1109-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 654895 INFO  (jetty-launcher-1061-thread-1) [n:127.0.0.1:36242_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 654899 INFO  (jetty-launcher-1061-thread-1) [n:127.0.0.1:36242_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 654899 INFO  (jetty-launcher-1061-thread-1) [n:127.0.0.1:36242_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36242_solr
   [junit4]   2> 654904 INFO  (jetty-launcher-1061-thread-3) [n:127.0.0.1:45836_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45836.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@110629b6
   [junit4]   2> 654906 INFO  (zkCallback-1087-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 654906 INFO  (zkCallback-1109-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 654906 INFO  (zkCallback-1085-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 654907 INFO  (zkCallback-1113-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 654909 INFO  (jetty-launcher-1061-thread-2) [n:127.0.0.1:43613_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 654917 INFO  (jetty-launcher-1061-thread-4) [n:127.0.0.1:44492_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 654946 INFO  (zkCallback-1095-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 654963 INFO  (jetty-launcher-1061-thread-3) [n:127.0.0.1:45836_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45836.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@110629b6
   [junit4]   2> 654963 INFO  (jetty-launcher-1061-thread-3) [n:127.0.0.1:45836_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45836.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@110629b6
   [junit4]   2> 654964 INFO  (jetty-launcher-1061-thread-3) [n:127.0.0.1:45836_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node3/.
   [junit4]   2> 654973 INFO  (jetty-launcher-1061-thread-2) [n:127.0.0.1:43613_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 654973 INFO  (jetty-launcher-1061-thread-2) [n:127.0.0.1:43613_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43613_solr
   [junit4]   2> 654974 INFO  (zkCallback-1109-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 654974 INFO  (zkCallback-1113-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 654974 INFO  (zkCallback-1087-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 654975 INFO  (jetty-launcher-1061-thread-4) [n:127.0.0.1:44492_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44492.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@110629b6
   [junit4]   2> 654982 INFO  (jetty-launcher-1061-thread-4) [n:127.0.0.1:44492_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44492.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@110629b6
   [junit4]   2> 654982 INFO  (jetty-launcher-1061-thread-4) [n:127.0.0.1:44492_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44492.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@110629b6
   [junit4]   2> 654983 INFO  (jetty-launcher-1061-thread-4) [n:127.0.0.1:44492_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/.
   [junit4]   2> 654990 INFO  (zkConnectionManagerCallback-1121-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 654991 INFO  (jetty-launcher-1061-thread-1) [n:127.0.0.1:36242_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 654998 INFO  (zkCallback-1085-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 654998 INFO  (zkCallback-1095-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 654998 INFO  (jetty-launcher-1061-thread-1) [n:127.0.0.1:36242_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44910/solr ready
   [junit4]   2> 654999 INFO  (jetty-launcher-1061-thread-1) [n:127.0.0.1:36242_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 655006 INFO  (zkCallback-1089-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 655011 INFO  (zkConnectionManagerCallback-1127-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 655012 INFO  (jetty-launcher-1061-thread-2) [n:127.0.0.1:43613_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 655013 INFO  (jetty-launcher-1061-thread-2) [n:127.0.0.1:43613_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44910/solr ready
   [junit4]   2> 655014 INFO  (jetty-launcher-1061-thread-2) [n:127.0.0.1:43613_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 655015 INFO  (jetty-launcher-1061-thread-1) [n:127.0.0.1:36242_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36242.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@110629b6
   [junit4]   2> 655027 INFO  (jetty-launcher-1061-thread-2) [n:127.0.0.1:43613_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43613.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@110629b6
   [junit4]   2> 655038 INFO  (jetty-launcher-1061-thread-2) [n:127.0.0.1:43613_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43613.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@110629b6
   [junit4]   2> 655038 INFO  (jetty-launcher-1061-thread-2) [n:127.0.0.1:43613_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43613.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@110629b6
   [junit4]   2> 655039 INFO  (jetty-launcher-1061-thread-2) [n:127.0.0.1:43613_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node2/.
   [junit4]   2> 655040 INFO  (jetty-launcher-1061-thread-1) [n:127.0.0.1:36242_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36242.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@110629b6
   [junit4]   2> 655040 INFO  (jetty-launcher-1061-thread-1) [n:127.0.0.1:36242_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36242.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@110629b6
   [junit4]   2> 655041 INFO  (jetty-launcher-1061-thread-1) [n:127.0.0.1:36242_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node1/.
   [junit4]   2> 655107 INFO  (zkConnectionManagerCallback-1130-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 655149 INFO  (TEST-TimeRoutedAliasUpdateProcessorTest.test-seed#[E3C53B70919B912C]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 655172 INFO  (qtp1661342372-4179) [n:127.0.0.1:45836_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=CLUSTERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 655177 INFO  (qtp1661342372-4179) [n:127.0.0.1:45836_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=4
   [junit4]   2> 655179 INFO  (TEST-TimeRoutedAliasUpdateProcessorTest.test-seed#[E3C53B70919B912C]) [    ] o.a.s.u.p.TimeRoutedAliasUpdateProcessorTest SolrClient: org.apache.solr.client.solrj.impl.CloudSolrClient@c39056a
   [junit4]   2> 655179 INFO  (TEST-TimeRoutedAliasUpdateProcessorTest.test-seed#[E3C53B70919B912C]) [    ] o.a.s.u.p.TimeRoutedAliasUpdateProcessorTest ClusterStateProvider org.apache.solr.client.solrj.impl.HttpClusterStateProvider@b1d798c
   [junit4]   2> 655190 INFO  (qtp1661342372-4181) [n:127.0.0.1:45836_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=CLUSTERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 655191 INFO  (qtp1661342372-4181) [n:127.0.0.1:45836_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 655192 INFO  (qtp1661342372-4182) [n:127.0.0.1:45836_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=CLUSTERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 655192 INFO  (qtp1661342372-4182) [n:127.0.0.1:45836_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 655193 INFO  (qtp1661342372-4181) [n:127.0.0.1:45836_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=CLUSTERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 655193 INFO  (qtp1661342372-4181) [n:127.0.0.1:45836_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 655194 INFO  (qtp1661342372-4182) [n:127.0.0.1:45836_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=CLUSTERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 655195 INFO  (qtp1661342372-4182) [n:127.0.0.1:45836_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 655198 INFO  (qtp1661342372-4179) [n:127.0.0.1:45836_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :create with params baseConfigSet=_default&name=timeConfigtest&action=CREATE&wt=javabin&version=2
   [junit4]   2> 655199 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/collection-queue-work
   [junit4]   2> 655199 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/overseer/collection-queue-work/qnr-0000000000
   [junit4]   2> 655199 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 655199 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.ZkDistributedQueue Found child node with improper name: qnr-0000000000
   [junit4]   2> 655199 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/collection-queue-work
   [junit4]   2> 655200 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 655200 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.ZkDistributedQueue Found child node with improper name: qnr-0000000000
   [junit4]   2> 655200 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskQueue Returning topN elements: [/overseer/collection-queue-work/qn-0000000000, ]
   [junit4]   2> 655200 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor Got 1 tasks from work-queue : [[org.apache.solr.cloud.OverseerTaskQueue$QueueEvent@2e684ecf]]
   [junit4]   2> 655200 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor Marked task [/overseer/collection-queue-work/qn-0000000000] as running
   [junit4]   2> 655200 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2>   "name":"timeConfigtest",
   [junit4]   2>   "baseConfigSet":"_default",
   [junit4]   2>   "immutable":"false",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 655201 DEBUG (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor Runner processing /overseer/collection-queue-work/qn-0000000000
   [junit4]   2> 655201 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"timeConfigtest",
   [junit4]   2>   "baseConfigSet":"_default",
   [junit4]   2>   "immutable":"false",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 655201 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor Cleaning up work-queue. #Running tasks: 1
   [junit4]   2> 655201 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor RunningTasks: [/overseer/collection-queue-work/qn-0000000000]
   [junit4]   2> 655201 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor BlockedTasks: []
   [junit4]   2> 655201 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor CompletedTasks: []
   [junit4]   2> 655201 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor RunningZKTasks: [/overseer/collection-queue-work/qn-0000000000]
   [junit4]   2> 655201 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskQueue Peeking for top 99 elements. ExcludeSet: [/overseer/collection-queue-work/qn-0000000000],[]
   [junit4]   2> 655201 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/protwords.txt to /configs/timeConfigtest/protwords.txt
   [junit4]   2> 655203 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/managed-schema to /configs/timeConfigtest/managed-schema
   [junit4]   2> 655205 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/solrconfig.xml to /configs/timeConfigtest/solrconfig.xml
   [junit4]   2> 655207 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/synonyms.txt to /configs/timeConfigtest/synonyms.txt
   [junit4]   2> 655208 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/stopwords.txt to /configs/timeConfigtest/stopwords.txt
   [junit4]   2> 655209 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/params.json to /configs/timeConfigtest/params.json
   [junit4]   2> 655211 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_it.txt to /configs/timeConfigtest/lang/contractions_it.txt
   [junit4]   2> 655213 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_no.txt to /configs/timeConfigtest/lang/stopwords_no.txt
   [junit4]   2> 655215 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hy.txt to /configs/timeConfigtest/lang/stopwords_hy.txt
   [junit4]   2> 655216 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stemdict_nl.txt to /configs/timeConfigtest/lang/stemdict_nl.txt
   [junit4]   2> 655218 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ca.txt to /configs/timeConfigtest/lang/contractions_ca.txt
   [junit4]   2> 655220 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_id.txt to /configs/timeConfigtest/lang/stopwords_id.txt
   [junit4]   2> 655222 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ga.txt to /configs/timeConfigtest/lang/stopwords_ga.txt
   [junit4]   2> 655223 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_da.txt to /configs/timeConfigtest/lang/stopwords_da.txt
   [junit4]   2> 655225 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hi.txt to /configs/timeConfigtest/lang/stopwords_hi.txt
   [junit4]   2> 655227 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ja.txt to /configs/timeConfigtest/lang/stopwords_ja.txt
   [junit4]   2> 655228 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_pt.txt to /configs/timeConfigtest/lang/stopwords_pt.txt
   [junit4]   2> 655230 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hu.txt to /configs/timeConfigtest/lang/stopwords_hu.txt
   [junit4]   2> 655232 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_el.txt to /configs/timeConfigtest/lang/stopwords_el.txt
   [junit4]   2> 655233 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ru.txt to /configs/timeConfigtest/lang/stopwords_ru.txt
   [junit4]   2> 655234 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_tr.txt to /configs/timeConfigtest/lang/stopwords_tr.txt
   [junit4]   2> 655236 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ar.txt to /configs/timeConfigtest/lang/stopwords_ar.txt
   [junit4]   2> 655237 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/userdict_ja.txt to /configs/timeConfigtest/lang/userdict_ja.txt
   [junit4]   2> 655239 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_it.txt to /configs/timeConfigtest/lang/stopwords_it.txt
   [junit4]   2> 655240 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_eu.txt to /configs/timeConfigtest/lang/stopwords_eu.txt
   [junit4]   2> 655242 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_cz.txt to /configs/timeConfigtest/lang/stopwords_cz.txt
   [junit4]   2> 655243 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stoptags_ja.txt to /configs/timeConfigtest/lang/stoptags_ja.txt
   [junit4]   2> 655245 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_fr.txt to /configs/timeConfigtest/lang/contractions_fr.txt
   [junit4]   2> 655246 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fa.txt to /configs/timeConfigtest/lang/stopwords_fa.txt
   [junit4]   2> 655248 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_de.txt to /configs/timeConfigtest/lang/stopwords_de.txt
   [junit4]   2> 655249 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ga.txt to /configs/timeConfigtest/lang/contractions_ga.txt
   [junit4]   2> 655251 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ca.txt to /configs/timeConfigtest/lang/stopwords_ca.txt
   [junit4]   2> 655252 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_nl.txt to /configs/timeConfigtest/lang/stopwords_nl.txt
   [junit4]   2> 655254 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_bg.txt to /configs/timeConfigtest/lang/stopwords_bg.txt
   [junit4]   2> 655255 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_es.txt to /configs/timeConfigtest/lang/stopwords_es.txt
   [junit4]   2> 655257 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_en.txt to /configs/timeConfigtest/lang/stopwords_en.txt
   [junit4]   2> 655258 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_lv.txt to /configs/timeConfigtest/lang/stopwords_lv.txt
   [junit4]   2> 655260 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_th.txt to /configs/timeConfigtest/lang/stopwords_th.txt
   [junit4]   2> 655261 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ro.txt to /configs/timeConfigtest/lang/stopwords_ro.txt
   [junit4]   2> 655263 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fr.txt to /configs/timeConfigtest/lang/stopwords_fr.txt
   [junit4]   2> 655265 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_sv.txt to /configs/timeConfigtest/lang/stopwords_sv.txt
   [junit4]   2> 655266 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fi.txt to /configs/timeConfigtest/lang/stopwords_fi.txt
   [junit4]   2> 655268 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/hyphenations_ga.txt to /configs/timeConfigtest/lang/hyphenations_ga.txt
   [junit4]   2> 655269 INFO  (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_gl.txt to /configs/timeConfigtest/lang/stopwords_gl.txt
   [junit4]   2> 655279 DEBUG (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor Completed task:[/overseer/collection-queue-work/qn-0000000000]
   [junit4]   2> 655280 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: data: /solr/overseer/collection-queue-work/qnr-0000000000
   [junit4]   2> 655280 DEBUG (zkCallback-1087-thread-1) [    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000000 state SyncConnected latchEventType null
   [junit4]   2> 655280 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/collection-queue-work
   [junit4]   2> 655280 DEBUG (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor Marked task [/overseer/collection-queue-work/qn-0000000000] as completed.
   [junit4]   2> 655280 DEBUG (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor RunningTasks: []
   [junit4]   2> 655281 DEBUG (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor BlockedTasks: []
   [junit4]   2> 655281 DEBUG (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor CompletedTasks: [/overseer/collection-queue-work/qn-0000000000]
   [junit4]   2> 655281 DEBUG (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor RunningZKTasks: [/overseer/collection-queue-work/qn-0000000000]
   [junit4]   2> 655281 DEBUG (OverseerThreadFactory-1778-thread-1-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000000 complete, response:{}
   [junit4]   2> 655290 INFO  (qtp1661342372-4179) [n:127.0.0.1:45836_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/configs params={baseConfigSet=_default&name=timeConfigtest&action=CREATE&wt=javabin&version=2} status=0 QTime=92
   [junit4]   2> 655291 INFO  (qtp1661342372-4181) [n:127.0.0.1:45836_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=CLUSTERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 655298 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 655299 INFO  (qtp1661342372-4181) [n:127.0.0.1:45836_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=7
   [junit4]   2> 655300 INFO  (qtp1661342372-4179) [n:127.0.0.1:45836_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=CLUSTERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 655300 INFO  (qtp1661342372-4179) [n:127.0.0.1:45836_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 655301 INFO  (qtp1661342372-4181) [n:127.0.0.1:45836_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=CLUSTERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 655301 INFO  (qtp1661342372-4181) [n:127.0.0.1:45836_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 655303 INFO  (qtp1661342372-4179) [n:127.0.0.1:45836_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=CLUSTERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 655304 INFO  (qtp1661342372-4179) [n:127.0.0.1:45836_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 655304 INFO  (qtp1661342372-4182) [n:127.0.0.1:45836_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=timeConfigtest&name=timeConfigtest&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 655305 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/collection-queue-work
   [junit4]   2> 655305 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/overseer/collection-queue-work/qnr-0000000002
   [junit4]   2> 655305 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 655306 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.ZkDistributedQueue Found child node with improper name: qnr-0000000002
   [junit4]   2> 655306 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/collection-queue-work
   [junit4]   2> 655306 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 655314 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.ZkDistributedQueue Found child node with improper name: qnr-0000000002
   [junit4]   2> 655315 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskQueue Returning topN elements: [/overseer/collection-queue-work/qn-0000000002, ]
   [junit4]   2> 655315 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor Got 1 tasks from work-queue : [[org.apache.solr.cloud.OverseerTaskQueue$QueueEvent@2e684ed1]]
   [junit4]   2> 655315 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor Marked task [/overseer/collection-queue-work/qn-0000000002] as running
   [junit4]   2> 655315 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000002 message:{
   [junit4]   2>   "name":"timeConfigtest",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "collection.configName":"timeConfigtest",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "nrtReplicas":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 655315 DEBUG (OverseerThreadFactory-1778-thread-2-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor Runner processing /overseer/collection-queue-work/qn-0000000002
   [junit4]   2> 655315 DEBUG (OverseerThreadFactory-1778-thread-2-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.a.c.OverseerCollectionMessageHandler OverseerCollectionMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"timeConfigtest",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "collection.configName":"timeConfigtest",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "nrtReplicas":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 655315 INFO  (OverseerThreadFactory-1778-thread-2-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection timeConfigtest
   [junit4]   2> 655315 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor Cleaning up work-queue. #Running tasks: 1
   [junit4]   2> 655315 INFO  (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 655316 DEBUG (OverseerThreadFactory-1778-thread-2-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.a.c.OverseerCollectionMessageHandler creating collections conf node /collections/timeConfigtest
   [junit4]   2> 655316 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor RunningTasks: [/overseer/collection-queue-work/qn-0000000002]
   [junit4]   2> 655316 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor BlockedTasks: []
   [junit4]   2> 655316 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor CompletedTasks: []
   [junit4]   2> 655316 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskProcessor RunningZKTasks: [/overseer/collection-queue-work/qn-0000000002]
   [junit4]   2> 655316 DEBUG (OverseerCollectionConfigSetProcessor-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.OverseerTaskQueue Peeking for top 99 elements. ExcludeSet: [/overseer/collection-queue-work/qn-0000000002],[]
   [junit4]   2> 655316 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/collections
   [junit4]   2> 655316 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/collections
   [junit4]   2> 655316 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/collections
   [junit4]   2> 655317 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/collections
   [junit4]   2> 655317 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/collections
   [junit4]   2> 655317 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/collections
   [junit4]   2> 655317 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/collections
   [junit4]   2> 655317 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/collections
   [junit4]   2> 655317 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 655317 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 655317 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 655317 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 655317 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 655317 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 655318 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 655318 DEBUG (OverseerThreadFactory-1778-thread-2-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.a.c.CreateCollectionCmd Check for collection zkNode:timeConfigtest
   [junit4]   2> 655318 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 655318 DEBUG (OverseerThreadFactory-1778-thread-2-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.a.c.CreateCollectionCmd Collection zkNode exists
   [junit4]   2> 655319 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 655319 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 655320 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 655321 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 655420 DEBUG (OverseerThreadFactory-1778-thread-2-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.a.c.Assign Identify nodes using default
   [junit4]   2> 655421 DEBUG (OverseerThreadFactory-1778-thread-2-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.a.c.CreateCollectionCmd Creating SolrCores for new collection timeConfigtest, shardNames [shard1] , message : {
   [junit4]   2>   "name":"timeConfigtest",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "collection.configName":"timeConfigtest",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "nrtReplicas":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 655422 DEBUG (OverseerThreadFactory-1778-thread-2-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.a.c.CreateCollectionCmd Creating core timeConfigtest_shard1_replica_n1 as part of shard shard1 of collection timeConfigtest on 127.0.0.1:44492_solr
   [junit4]   2> 655423 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 655423 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 655424 DEBUG (OverseerThreadFactory-1778-thread-2-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.a.c.OverseerCollectionMessageHandler Expecting [timeConfigtest_shard1_replica_n1] cores but found {}
   [junit4]   2> 655424 INFO  (OverseerStateUpdate-73323413769158668-127.0.0.1:44492_solr-n_0000000000) [n:127.0.0.1:44492_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"timeConfigtest",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"timeConfigtest_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:44492/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 655524 DEBUG (OverseerThreadFactory-1778-thread-2-processing-n:127.0.0.1:44492_solr) [n:127.0.0.1:44492_solr    ] o.a.s.c.a.c.OverseerCollectionMessageHandler Expecting [timeConfigtest_shard1_replica_n1] cores but found {}
   [junit4]   2> 655526 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 655527 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 655643 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr    x:timeConfigtest_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=timeConfigtest&newCollection=true&name=timeConfigtest_shard1_replica_n1&action=CREATE&numShards=1&collection=timeConfigtest&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 655645 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/collections/timeConfigtest/state.json
   [junit4]   2> 655645 DEBUG (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 655646 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 655646 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 655646 DEBUG (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.ZkController Registering watch for collection timeConfigtest
   [junit4]   2> 655646 DEBUG (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.ZkController waiting to find shard id in clusterstate for timeConfigtest_shard1_replica_n1
   [junit4]   2> 655747 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: data: /solr/collections/timeConfigtest/state.json
   [junit4]   2> 655748 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 655750 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 656658 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
   [junit4]   2> 656659 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.RequestParams request params refreshed to version 0
   [junit4]   2> 656659 WARN  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../contrib/extraction/lib filtered by .*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../contrib/extraction/lib
   [junit4]   2> 656659 WARN  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../dist filtered by solr-cell-\d.*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../dist
   [junit4]   2> 656660 WARN  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../contrib/clustering/lib filtered by .*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../contrib/clustering/lib
   [junit4]   2> 656660 WARN  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../dist filtered by solr-clustering-\d.*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../dist
   [junit4]   2> 656660 WARN  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../contrib/langid/lib filtered by .*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../contrib/langid/lib
   [junit4]   2> 656660 WARN  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../dist filtered by solr-langid-\d.*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../dist
   [junit4]   2> 656660 WARN  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../contrib/velocity/lib filtered by .*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../contrib/velocity/lib
   [junit4]   2> 656660 WARN  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../dist filtered by solr-velocity-\d.*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../dist
   [junit4]   2> 656661 WARN  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../dist filtered by solr-ltr-\d.*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1/../../../../dist
   [junit4]   2> 656667 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 656685 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.s.IndexSchema [timeConfigtest_shard1_replica_n1] Schema name=default-config
   [junit4]   2> 656842 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
   [junit4]   2> 656883 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'timeConfigtest_shard1_replica_n1' using configuration from collection timeConfigtest, trusted=true
   [junit4]   2> 656883 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44492.solr.core.timeConfigtest.shard1.replica_n1' (registry 'solr.core.timeConfigtest.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@110629b6
   [junit4]   2> 656884 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.SolrCore [[timeConfigtest_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/timeConfigtest_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E3C53B70919B912C-001/tempDir-001/node4/./timeConfigtest_shard1_replica_n1/data/]
   [junit4]   2> 656887 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
   [junit4]   2> 656953 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 656953 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 656955 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
   [junit4]   2> 656955 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 656957 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@589a39cf[timeConfigtest_shard1_replica_n1] main]
   [junit4]   2> 656958 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/timeConfigtest
   [junit4]   2> 656958 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/timeConfigtest
   [junit4]   2> 656958 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/timeConfigtest/managed-schema
   [junit4]   2> 656958 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/configs/timeConfigtest/managed-schema
   [junit4]   2> 656958 DEBUG (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.ZkController Added new OnReconnect listener ZkIndexSchemaReader: /configs/timeConfigtest/managed-schema, uniqueCoreId: timeConfigtest_shard1_replica_n1:19314879025382350
   [junit4]   2> 656970 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 656971 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 656971 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
   [junit4]   2> 656973 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 656973 INFO  (searcherExecutor-1792-thread-1-processing-n:127.0.0.1:44492_solr x:timeConfigtest_shard1_replica_n1 c:timeConfigtest s:shard1 r:core_node2) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@589a39cf[timeConfigtest_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 656973 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1611637696157974528
   [junit4]   2> 656973 INFO  (searcherExecutor-1792-thread-1-processing-n:127.0.0.1:44492_solr x:timeConfigtest_shard1_replica_n1 c:timeConfigtest s:shard1 r:core_node2) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener done.
   [junit4]   2> 656973 INFO  (searcherExecutor-1792-thread-1-processing-n:127.0.0.1:44492_solr x:timeConfigtest_shard1_replica_n1 c:timeConfigtest s:shard1 r:core_node2) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 656974 INFO  (searcherExecutor-1792-thread-1-processing-n:127.0.0.1:44492_solr x:timeConfigtest_shard1_replica_n1 c:timeConfigtest s:shard1 r:core_node2) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.SolrCore [timeConfigtest_shard1_replica_n1] Registered new searcher Searcher@589a39cf[timeConfigtest_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 656979 DEBUG (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.ZkController watch zkdir /configs/timeConfigtest
   [junit4]   2> 656979 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/configs/timeConfigtest
   [junit4]   2> 656980 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/collections/timeConfigtest/state.json
   [junit4]   2> 656982 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/collections/timeConfigtest/terms/shard1
   [junit4]   2> 656983 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: data: /solr/collections/timeConfigtest/terms/shard1
   [junit4]   2> 656983 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/timeConfigtest/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 656983 DEBUG (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.ZkController Register replica - core:timeConfigtest_shard1_replica_n1 address:https://127.0.0.1:44492/solr collection:timeConfigtest shard:shard1
   [junit4]   2> 656983 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/collections/timeConfigtest/terms/shard1
   [junit4]   2> 656985 DEBUG (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/timeConfigtest/leader_elect/shard1/election/73323413769158668-core_node2-n_0000000000
   [junit4]   2> 656985 DEBUG (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=10000
   [junit4]   2> 656986 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 656986 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 656986 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:44492/solr/timeConfigtest_shard1_replica_n1/
   [junit4]   2> 656986 INFO  (qtp1036933611-4215) [n:127.0.0.1:44492_solr c:timeConfigtest s:shard1 r:core_node2 x:timeConfigtest_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 656987 INFO  (qtp1036933611-4215) [n:

[...truncated too long message...]

ding settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null590874220
     [copy] Copying 238 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null590874220
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null590874220

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 34ms :: artifacts dl 5ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 598 minutes 20 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-NightlyTests-master - Build # 29 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-master/29/

4 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimExecutePlanAction.testIntegration

Error Message:
Timed out waiting for replicas of new collection to be active Live Nodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr] Last available state: DocCollection(testIntegration//clusterstate.json/4)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{"shard1":{       "replicas":{         "core_node1":{           "core":"testIntegration_shard1_replica_n1",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10000_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node2":{           "core":"testIntegration_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10001_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"80000000-7fffffff",       "state":"active"}}}

Stack Trace:
java.lang.AssertionError: Timed out waiting for replicas of new collection to be active
Live Nodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
Last available state: DocCollection(testIntegration//clusterstate.json/4)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{"shard1":{
      "replicas":{
        "core_node1":{
          "core":"testIntegration_shard1_replica_n1",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10000_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node2":{
          "core":"testIntegration_shard1_replica_n2",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10001_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active"}}}
        at __randomizedtesting.SeedInfo.seed([C053896FDCF92068:70328743F9C6814D]:0)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:70)
        at org.apache.solr.cloud.autoscaling.sim.TestSimExecutePlanAction.testIntegration(TestSimExecutePlanAction.java:181)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: last state: DocCollection(testIntegration//clusterstate.json/4)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{"shard1":{
      "replicas":{
        "core_node1":{
          "core":"testIntegration_shard1_replica_n1",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10000_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node2":{
          "core":"testIntegration_shard1_replica_n2",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10001_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active"}}}
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:112)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:64)
        ... 40 more


FAILED:  org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testDeleteNode

Error Message:
unexpected DELETENODE status: {responseHeader={status=0,QTime=71},status={state=notfound,msg=Did not find [search_rate_trigger3/471b5fbc1fbd22T855pzt2qu6y3pn5kq7ex5gjyb/0] in any tasks queue}}

Stack Trace:
java.lang.AssertionError: unexpected DELETENODE status: {responseHeader={status=0,QTime=71},status={state=notfound,msg=Did not find [search_rate_trigger3/471b5fbc1fbd22T855pzt2qu6y3pn5kq7ex5gjyb/0] in any tasks queue}}
        at __randomizedtesting.SeedInfo.seed([C053896FDCF92068:E2C147EDEB33AF15]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.lambda$testDeleteNode$5(SearchRateTriggerIntegrationTest.java:684)
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testDeleteNode(SearchRateTriggerIntegrationTest.java:676)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testListeners

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
        at __randomizedtesting.SeedInfo.seed([C053896FDCF92068:2D931E7F6195BDB1]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testListeners(TestSimTriggerIntegration.java:1001)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsBasicDistributedZk2Test.test

Error Message:
Doc Counts do not add up expected:<67> but was:<36>

Stack Trace:
java.lang.AssertionError: Doc Counts do not add up expected:<67> but was:<36>
        at __randomizedtesting.SeedInfo.seed([C053896FDCF92068:4807B6B572054D90]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.assertDocCounts(AbstractFullDistribZkTestBase.java:1453)
        at org.apache.solr.cloud.BasicDistributedZk2Test.test(BasicDistributedZk2Test.java:101)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1008)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:983)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14059 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsBasicDistributedZk2Test
   [junit4]   2> 2712405 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/init-core-data-001
   [junit4]   2> 2712405 WARN  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 2712406 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2712408 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 2712408 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 2712470 WARN  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 2712482 WARN  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2712484 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2712496 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_lucene2.us.west_apache_org_35839_hdfs____bpmx5y/webapp
   [junit4]   2> 2712849 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.m.log Started HttpServer2$[hidden email]:35839
   [junit4]   2> 2712936 WARN  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2712937 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2712950 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_36607_datanode____.fb0g4f/webapp
   [junit4]   2> 2713275 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36607
   [junit4]   2> 2713282 WARN  (1157835137@qtp-68981018-0 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36607) [    ] o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is false. Rechecking.
   [junit4]   2> 2713292 WARN  (1157835137@qtp-68981018-0 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36607) [    ] o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is true
   [junit4]   2> 2713318 WARN  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2713318 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2713330 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_45060_datanode____.87mua8/webapp
   [junit4]   2> 2713411 ERROR (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:36530) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 2713420 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x4716a8fc2cede5: from storage DS-ac956d56-734d-4ae1-81cf-dbf6d8248c3b node DatanodeRegistration(127.0.0.1:46272, datanodeUuid=ee4bf1e7-402f-4175-b9ce-910dca7ea09a, infoPort=35271, infoSecurePort=0, ipcPort=38647, storageInfo=lv=-56;cid=testClusterID;nsid=1886280475;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 2713420 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x4716a8fc2cede5: from storage DS-e97dd61a-7f57-4b5c-a0bd-6f20df120b45 node DatanodeRegistration(127.0.0.1:46272, datanodeUuid=ee4bf1e7-402f-4175-b9ce-910dca7ea09a, infoPort=35271, infoSecurePort=0, ipcPort=38647, storageInfo=lv=-56;cid=testClusterID;nsid=1886280475;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 2713688 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[C053896FDCF92068]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45060
   [junit4]   2> 2713779 ERROR (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:36530) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 2713798 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x4716a9125c3b12: from storage DS-4e3d1460-6f13-4d76-b1fb-5701eac39984 node DatanodeRegistration(127.0.0.1:44026, datanodeUuid=0a360f97-66db-4769-aedc-23cec655f904, infoPort=43076, infoSecurePort=0, ipcPort=45408, storageInfo=lv=-56;cid=testClusterID;nsid=1886280475;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 2713798 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x4716a9125c3b12: from storage DS-6149dd71-4972-4001-a6bb-d3cdaa34c389 node DatanodeRegistration(127.0.0.1:44026, datanodeUuid=0a360f97-66db-4769-aedc-23cec655f904, infoPort=43076, infoSecurePort=0, ipcPort=45408, storageInfo=lv=-56;cid=testClusterID;nsid=1886280475;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 2713860 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2713861 INFO  (Thread-6220) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2713861 INFO  (Thread-6220) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2713862 ERROR (Thread-6220) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2713961 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.ZkTestServer start zk server on port:34867
   [junit4]   2> 2713964 INFO  (zkConnectionManagerCallback-4490-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2713968 INFO  (zkConnectionManagerCallback-4492-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2713972 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2713974 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2713975 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2713976 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2713978 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2713979 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2713980 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2713981 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2713982 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2713983 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2713985 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2713986 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use TLOG replicas unless explicitly asked otherwise
   [junit4]   2> 2714093 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2714094 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2714094 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2714094 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2714094 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@18c98332{/,null,AVAILABLE}
   [junit4]   2> 2714095 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3c29c13{HTTP/1.1,[http/1.1]}{127.0.0.1:35402}
   [junit4]   2> 2714096 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.Server Started @2714163ms
   [junit4]   2> 2714096 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:36530/hdfs__lucene2-us-west.apache.org_36530__home_jenkins_jenkins-slave_workspace_Lucene-Solr-BadApples-NightlyTests-master_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001_tempDir-002_control_data, replicaType=NRT, hostContext=/, hostPort=35402, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/control-001/cores}
   [junit4]   2> 2714096 ERROR (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2714096 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2714097 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2714097 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2714097 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2714097 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-23T03:10:38.130Z
   [junit4]   2> 2714098 INFO  (zkConnectionManagerCallback-4494-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2714099 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2714099 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/control-001/solr.xml
   [junit4]   2> 2714103 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2714103 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2714104 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@406fb424, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2714268 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34867/solr
   [junit4]   2> 2714270 INFO  (zkConnectionManagerCallback-4498-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2714271 INFO  (zkConnectionManagerCallback-4500-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2714353 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [n:127.0.0.1:35402_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35402_
   [junit4]   2> 2714354 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [n:127.0.0.1:35402_    ] o.a.s.c.Overseer Overseer (id=73368945728421892-127.0.0.1:35402_-n_0000000000) starting
   [junit4]   2> 2714362 INFO  (zkConnectionManagerCallback-4507-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2714363 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [n:127.0.0.1:35402_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34867/solr ready
   [junit4]   2> 2714364 INFO  (OverseerStateUpdate-73368945728421892-127.0.0.1:35402_-n_0000000000) [n:127.0.0.1:35402_    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:35402_
   [junit4]   2> 2714364 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [n:127.0.0.1:35402_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35402_
   [junit4]   2> 2714385 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [n:127.0.0.1:35402_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2714390 INFO  (zkCallback-4499-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2714390 INFO  (zkCallback-4506-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2714404 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [n:127.0.0.1:35402_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@406fb424
   [junit4]   2> 2714414 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [n:127.0.0.1:35402_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@406fb424
   [junit4]   2> 2714414 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [n:127.0.0.1:35402_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@406fb424
   [junit4]   2> 2714415 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [n:127.0.0.1:35402_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/control-001/cores
   [junit4]   2> 2714443 INFO  (zkConnectionManagerCallback-4513-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2714444 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2714445 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34867/solr ready
   [junit4]   2> 2714446 INFO  (qtp1435213711-18757) [n:127.0.0.1:35402_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:35402_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2714448 INFO  (OverseerThreadFactory-6102-thread-1-processing-n:127.0.0.1:35402_) [n:127.0.0.1:35402_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 2714556 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2714556 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2715574 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2715586 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2715691 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2715718 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 2715719 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@406fb424
   [junit4]   2> 2715719 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:36530/solr_hdfs_home
   [junit4]   2> 2715719 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 2715719 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[hdfs://lucene2-us-west.apache.org:36530/solr_hdfs_home/control_collection/core_node2/data/]
   [junit4]   2> 2715720 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:36530/solr_hdfs_home/control_collection/core_node2/data/snapshot_metadata
   [junit4]   2> 2715739 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 2715739 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 2715740 WARN  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 2720741 WARN  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 2725747 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 2725748 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:36530/solr_hdfs_home/control_collection/core_node2/data
   [junit4]   2> 2725770 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:36530/solr_hdfs_home/control_collection/core_node2/data/index
   [junit4]   2> 2725776 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 2725776 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 2725781 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 2725781 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=37, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=84.4853515625, floorSegmentMB=1.30859375, forceMergeDeletesPctAllowed=14.19908164066474, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=32.91266254187702
   [junit4]   2> 2725798 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:44026 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ac956d56-734d-4ae1-81cf-dbf6d8248c3b:NORMAL:127.0.0.1:46272|RBW], ReplicaUC[[DISK]DS-4e3d1460-6f13-4d76-b1fb-5701eac39984:NORMAL:127.0.0.1:44026|RBW]]} size 0
   [junit4]   2> 2725799 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46272 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4e3d1460-6f13-4d76-b1fb-5701eac39984:NORMAL:127.0.0.1:44026|RBW], ReplicaUC[[DISK]DS-e97dd61a-7f57-4b5c-a0bd-6f20df120b45:NORMAL:127.0.0.1:46272|FINALIZED]]} size 0
   [junit4]   2> 2725804 WARN  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2725847 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 2725847 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2725847 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 2725857 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2725857 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2725859 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@65b8086f
   [junit4]   2> 2725889 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@29aeeff4[control_collection_shard1_replica_n1] main]
   [junit4]   2> 2725890 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2725890 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2725891 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2725892 INFO  (searcherExecutor-6104-thread-1-processing-n:127.0.0.1:35402_ x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@29aeeff4[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2725892 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1612366217137356800
   [junit4]   2> 2725900 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 2725904 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2725904 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2725904 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:35402/control_collection_shard1_replica_n1/
   [junit4]   2> 2725905 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2725905 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:35402/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 2725905 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2725911 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:35402/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 2725963 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2725965 INFO  (qtp1435213711-18752) [n:127.0.0.1:35402_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=11409
   [junit4]   2> 2725968 INFO  (qtp1435213711-18757) [n:127.0.0.1:35402_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2726066 INFO  (zkCallback-4499-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2726454 INFO  (OverseerCollectionConfigSetProcessor-73368945728421892-127.0.0.1:35402_-n_0000000000) [n:127.0.0.1:35402_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2726968 INFO  (qtp1435213711-18757) [n:127.0.0.1:35402_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:35402_&wt=javabin&version=2} status=0 QTime=12522
   [junit4]   2> 2726973 INFO  (zkConnectionManagerCallback-4518-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2726974 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2726975 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34867/solr ready
   [junit4]   2> 2726975 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2726976 INFO  (qtp1435213711-18753) [n:127.0.0.1:35402_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2726978 INFO  (OverseerThreadFactory-6102-thread-2-processing-n:127.0.0.1:35402_) [n:127.0.0.1:35402_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 2727182 WARN  (OverseerThreadFactory-6102-thread-2-processing-n:127.0.0.1:35402_) [n:127.0.0.1:35402_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 2727185 INFO  (qtp1435213711-18753) [n:127.0.0.1:35402_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2727185 INFO  (qtp1435213711-18753) [n:127.0.0.1:35402_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=209
   [junit4]   2> 2727291 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/shard-1-001 of type TLOG
   [junit4]   2> 2727292 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2727293 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2727293 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2727293 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2727293 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@41bca570{/,null,AVAILABLE}
   [junit4]   2> 2727295 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@f78a39{HTTP/1.1,[http/1.1]}{127.0.0.1:41276}
   [junit4]   2> 2727295 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.Server Started @2727362ms
   [junit4]   2> 2727295 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:36530/hdfs__lucene2-us-west.apache.org_36530__home_jenkins_jenkins-slave_workspace_Lucene-Solr-BadApples-NightlyTests-master_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001_tempDir-002_jetty1, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=41276, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/shard-1-001/cores}
   [junit4]   2> 2727295 ERROR (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2727296 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2727296 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2727296 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2727296 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2727296 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-23T03:10:51.329Z
   [junit4]   2> 2727297 INFO  (zkConnectionManagerCallback-4520-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2727298 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2727298 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/shard-1-001/solr.xml
   [junit4]   2> 2727302 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2727302 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2727303 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@406fb424, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2727438 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34867/solr
   [junit4]   2> 2727439 INFO  (zkConnectionManagerCallback-4524-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2727441 INFO  (zkConnectionManagerCallback-4526-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2727447 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2727451 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2727451 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41276_
   [junit4]   2> 2727452 INFO  (zkCallback-4499-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2727457 INFO  (zkCallback-4506-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2727457 INFO  (zkCallback-4517-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2727462 INFO  (zkCallback-4525-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2727473 INFO  (zkConnectionManagerCallback-4533-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2727474 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2727475 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34867/solr ready
   [junit4]   2> 2727476 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2727493 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@406fb424
   [junit4]   2> 2727503 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@406fb424
   [junit4]   2> 2727503 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@406fb424
   [junit4]   2> 2727505 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/shard-1-001/cores
   [junit4]   2> 2727530 INFO  (qtp1435213711-18759) [n:127.0.0.1:35402_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:41276_&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2727532 INFO  (OverseerCollectionConfigSetProcessor-73368945728421892-127.0.0.1:35402_-n_0000000000) [n:127.0.0.1:35402_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2727533 INFO  (OverseerThreadFactory-6102-thread-3-processing-n:127.0.0.1:35402_) [n:127.0.0.1:35402_ c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:41276_ for creating new replica of shard shard2
   [junit4]   2> 2727536 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_    x:collection1_shard2_replica_t41] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2728552 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2728564 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.IndexSchema [collection1_shard2_replica_t41] Schema name=test
   [junit4]   2> 2728679 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2728709 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_t41' using configuration from collection collection1, trusted=true
   [junit4]   2> 2728710 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_t41' (registry 'solr.core.collection1.shard2.replica_t41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@406fb424
   [junit4]   2> 2728710 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:36530/solr_hdfs_home
   [junit4]   2> 2728710 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 2728710 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SolrCore [[collection1_shard2_replica_t41] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/shard-1-001/cores/collection1_shard2_replica_t41], dataDir=[hdfs://lucene2-us-west.apache.org:36530/solr_hdfs_home/collection1/core_node42/data/]
   [junit4]   2> 2728711 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:36530/solr_hdfs_home/collection1/core_node42/data/snapshot_metadata
   [junit4]   2> 2728718 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 2728718 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 2728727 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 2728728 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:36530/solr_hdfs_home/collection1/core_node42/data
   [junit4]   2> 2728742 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:36530/solr_hdfs_home/collection1/core_node42/data/index
   [junit4]   2> 2728748 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 2728748 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 2728755 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 2728755 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=37, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=84.4853515625, floorSegmentMB=1.30859375, forceMergeDeletesPctAllowed=14.19908164066474, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=32.91266254187702
   [junit4]   2> 2728792 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:44026 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-e97dd61a-7f57-4b5c-a0bd-6f20df120b45:NORMAL:127.0.0.1:46272|RBW], ReplicaUC[[DISK]DS-6149dd71-4972-4001-a6bb-d3cdaa34c389:NORMAL:127.0.0.1:44026|RBW]]} size 0
   [junit4]   2> 2728793 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46272 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6149dd71-4972-4001-a6bb-d3cdaa34c389:NORMAL:127.0.0.1:44026|RBW], ReplicaUC[[DISK]DS-ac956d56-734d-4ae1-81cf-dbf6d8248c3b:NORMAL:127.0.0.1:46272|FINALIZED]]} size 0
   [junit4]   2> 2728796 WARN  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2728848 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 2728848 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2728848 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 2728857 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2728857 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2728859 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@7e869857
   [junit4]   2> 2728870 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.SolrIndexSearcher Opening [Searcher@4ff57eee[collection1_shard2_replica_t41] main]
   [junit4]   2> 2728871 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2728871 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2728872 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2728873 INFO  (searcherExecutor-6118-thread-1-processing-n:127.0.0.1:41276_ x:collection1_shard2_replica_t41 c:collection1 s:shard2) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SolrCore [collection1_shard2_replica_t41] Registered new searcher Searcher@4ff57eee[collection1_shard2_replica_t41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2728873 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1612366220263161856
   [junit4]   2> 2728879 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node42=0}, version=0}
   [junit4]   2> 2728881 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2728881 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2728881 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:41276/collection1_shard2_replica_t41/
   [junit4]   2> 2728882 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2728882 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SyncStrategy http://127.0.0.1:41276/collection1_shard2_replica_t41/ has no replicas
   [junit4]   2> 2728882 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2728882 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ZkController collection1_shard2_replica_t41 stopping background replication from leader
   [junit4]   2> 2728885 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:41276/collection1_shard2_replica_t41/ shard2
   [junit4]   2> 2728988 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2728989 INFO  (qtp1530925311-18823) [n:127.0.0.1:41276_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1453
   [junit4]   2> 2728992 INFO  (qtp1435213711-18759) [n:127.0.0.1:35402_ c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:41276_&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2} status=0 QTime=1461
   [junit4]   2> 2729090 INFO  (zkCallback-4525-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2729099 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/shard-2-001 of type TLOG
   [junit4]   2> 2729099 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2729100 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2729100 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2729100 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2729101 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@60f3cefe{/,null,AVAILABLE}
   [junit4]   2> 2729103 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1e111cf{HTTP/1.1,[http/1.1]}{127.0.0.1:40998}
   [junit4]   2> 2729103 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.e.j.s.Server Started @2729170ms
   [junit4]   2> 2729103 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:36530/hdfs__lucene2-us-west.apache.org_36530__home_jenkins_jenkins-slave_workspace_Lucene-Solr-BadApples-NightlyTests-master_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001_tempDir-002_jetty2, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=40998, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/shard-2-001/cores}
   [junit4]   2> 2729103 ERROR (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2729103 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2729103 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2729103 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2729103 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2729104 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-23T03:10:53.137Z
   [junit4]   2> 2729105 INFO  (zkConnectionManagerCallback-4536-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2729106 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2729106 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/shard-2-001/solr.xml
   [junit4]   2> 2729109 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2729109 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2729111 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@406fb424, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2729314 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34867/solr
   [junit4]   2> 2729315 INFO  (zkConnectionManagerCallback-4540-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2729317 INFO  (zkConnectionManagerCallback-4542-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2729323 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2729327 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2729327 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40998_
   [junit4]   2> 2729328 INFO  (zkCallback-4517-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2729328 INFO  (zkCallback-4506-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2729328 INFO  (zkCallback-4499-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2729328 INFO  (zkCallback-4525-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2729333 INFO  (zkCallback-4532-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2729333 INFO  (zkCallback-4541-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2729358 INFO  (zkConnectionManagerCallback-4549-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2729359 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2729360 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34867/solr ready
   [junit4]   2> 2729360 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2729378 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@406fb424
   [junit4]   2> 2729387 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@406fb424
   [junit4]   2> 2729388 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@406fb424
   [junit4]   2> 2729389 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[C053896FDCF92068]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_C053896FDCF92068-001/shard-2-001/cores
   [junit4]   2> 2729421 INFO  (qtp1435213711-18755) [n:127.0.0.1:35402_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:40998_&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2729423 INFO  (OverseerCollectionConfigSetProcessor-73368945728421892-127.0.0.1:35402_-n_0000000000) [n:127.0.0.1:35402_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2729425 INFO  (Over

[...truncated too long message...]

ing settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null975689639
     [copy] Copying 238 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null975689639
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null975689639

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 130ms :: artifacts dl 14ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 634 minutes 8 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-NightlyTests-master - Build # 30 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-master/30/

3 tests failed.
FAILED:  org.apache.solr.cloud.MoveReplicaHDFSTest.testFailedMove

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:36856/solr/MoveReplicaHDFSTest_failed_coll_true, http://127.0.0.1:35500/solr/MoveReplicaHDFSTest_failed_coll_true]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:36856/solr/MoveReplicaHDFSTest_failed_coll_true, http://127.0.0.1:35500/solr/MoveReplicaHDFSTest_failed_coll_true]
        at __randomizedtesting.SeedInfo.seed([408E7BC14D58FA08:EA43A833FA8B2FD8]:0)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:462)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:994)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.cloud.MoveReplicaTest.testFailedMove(MoveReplicaTest.java:301)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:36856/solr/MoveReplicaHDFSTest_failed_coll_true: no servers hosting shard: shard2
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        ... 46 more


FAILED:  org.apache.solr.cloud.TestAuthenticationFramework.testBasics

Error Message:
Error from server at http://127.0.0.1:45047/solr/testcollection_shard1_replica_n2: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/testcollection_shard1_replica_n2/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/testcollection_shard1_replica_n2/update. Reason: <pre>    Can not find: /solr/testcollection_shard1_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>  </body> </html>

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://127.0.0.1:45047/solr/testcollection_shard1_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/testcollection_shard1_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n2/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at __randomizedtesting.SeedInfo.seed([408E7BC14D58FA08:7D56D5ED75B6A478]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
        at org.apache.solr.cloud.TestAuthenticationFramework.collectionCreateSearchDeleteTwice(TestAuthenticationFramework.java:125)
        at org.apache.solr.cloud.TestAuthenticationFramework.testBasics(TestAuthenticationFramework.java:76)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:45047/solr/testcollection_shard1_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/testcollection_shard1_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n2/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testNodeLost

Error Message:
java.util.concurrent.TimeoutException: last ClusterState: znodeVersion: 307 live nodes:[127.0.0.1:10082_solr, 127.0.0.1:10004_solr, 127.0.0.1:10018_solr, 127.0.0.1:10049_solr, 127.0.0.1:10104_solr, 127.0.0.1:10020_solr, 127.0.0.1:10065_solr, 127.0.0.1:10021_solr, 127.0.0.1:10005_solr, 127.0.0.1:10066_solr, 127.0.0.1:10035_solr, 127.0.0.1:10019_solr, 127.0.0.1:10096_solr, 127.0.0.1:10022_solr, 127.0.0.1:10105_solr, 127.0.0.1:10047_solr, 127.0.0.1:10102_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10052_solr, 127.0.0.1:10006_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10003_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10101_solr, 127.0.0.1:10071_solr, 127.0.0.1:10023_solr, 127.0.0.1:10076_solr, 127.0.0.1:10007_solr, 127.0.0.1:10107_solr, 127.0.0.1:10055_solr, 127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10108_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10080_solr, 127.0.0.1:10064_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10012_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10013_solr, 127.0.0.1:10058_solr, 127.0.0.1:10090_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10009_solr, 127.0.0.1:10030_solr, 127.0.0.1:10011_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10015_solr, 127.0.0.1:10069_solr, 127.0.0.1:10110_solr, 127.0.0.1:10062_solr, 127.0.0.1:10008_solr, 127.0.0.1:10024_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10106_solr, 127.0.0.1:10100_solr, 127.0.0.1:10037_solr, 127.0.0.1:10109_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10103_solr, 127.0.0.1:10056_solr, 127.0.0.1:10072_solr] collections:{.system=DocCollection(.system//clusterstate.json/307)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{"shard1":{       "replicas":{},       "range":"80000000-7fffffff",       "state":"active"}}}}, last coll state: DocCollection(.system//clusterstate.json/307)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{"shard1":{       "replicas":{},       "range":"80000000-7fffffff",       "state":"active"}}}

Stack Trace:
java.io.IOException: java.util.concurrent.TimeoutException: last ClusterState: znodeVersion: 307
live nodes:[127.0.0.1:10082_solr, 127.0.0.1:10004_solr, 127.0.0.1:10018_solr, 127.0.0.1:10049_solr, 127.0.0.1:10104_solr, 127.0.0.1:10020_solr, 127.0.0.1:10065_solr, 127.0.0.1:10021_solr, 127.0.0.1:10005_solr, 127.0.0.1:10066_solr, 127.0.0.1:10035_solr, 127.0.0.1:10019_solr, 127.0.0.1:10096_solr, 127.0.0.1:10022_solr, 127.0.0.1:10105_solr, 127.0.0.1:10047_solr, 127.0.0.1:10102_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10052_solr, 127.0.0.1:10006_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10003_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10101_solr, 127.0.0.1:10071_solr, 127.0.0.1:10023_solr, 127.0.0.1:10076_solr, 127.0.0.1:10007_solr, 127.0.0.1:10107_solr, 127.0.0.1:10055_solr, 127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10108_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10080_solr, 127.0.0.1:10064_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10012_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10013_solr, 127.0.0.1:10058_solr, 127.0.0.1:10090_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10009_solr, 127.0.0.1:10030_solr, 127.0.0.1:10011_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10015_solr, 127.0.0.1:10069_solr, 127.0.0.1:10110_solr, 127.0.0.1:10062_solr, 127.0.0.1:10008_solr, 127.0.0.1:10024_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10106_solr, 127.0.0.1:10100_solr, 127.0.0.1:10037_solr, 127.0.0.1:10109_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10103_solr, 127.0.0.1:10056_solr, 127.0.0.1:10072_solr]
collections:{.system=DocCollection(.system//clusterstate.json/307)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{"shard1":{
      "replicas":{},
      "range":"80000000-7fffffff",
      "state":"active"}}}}, last coll state: DocCollection(.system//clusterstate.json/307)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{"shard1":{
      "replicas":{},
      "range":"80000000-7fffffff",
      "state":"active"}}}
        at __randomizedtesting.SeedInfo.seed([408E7BC14D58FA08:FF9BB53FCEB29F8E]:0)
        at org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.createSystemCollection(SimClusterStateProvider.java:1344)
        at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.setupTest(TestSimLargeCluster.java:137)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:969)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: last ClusterState: znodeVersion: 307
live nodes:[127.0.0.1:10082_solr, 127.0.0.1:10004_solr, 127.0.0.1:10018_solr, 127.0.0.1:10049_solr, 127.0.0.1:10104_solr, 127.0.0.1:10020_solr, 127.0.0.1:10065_solr, 127.0.0.1:10021_solr, 127.0.0.1:10005_solr, 127.0.0.1:10066_solr, 127.0.0.1:10035_solr, 127.0.0.1:10019_solr, 127.0.0.1:10096_solr, 127.0.0.1:10022_solr, 127.0.0.1:10105_solr, 127.0.0.1:10047_solr, 127.0.0.1:10102_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10052_solr, 127.0.0.1:10006_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10003_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10101_solr, 127.0.0.1:10071_solr, 127.0.0.1:10023_solr, 127.0.0.1:10076_solr, 127.0.0.1:10007_solr, 127.0.0.1:10107_solr, 127.0.0.1:10055_solr, 127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10108_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10080_solr, 127.0.0.1:10064_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10012_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10013_solr, 127.0.0.1:10058_solr, 127.0.0.1:10090_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10009_solr, 127.0.0.1:10030_solr, 127.0.0.1:10011_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10015_solr, 127.0.0.1:10069_solr, 127.0.0.1:10110_solr, 127.0.0.1:10062_solr, 127.0.0.1:10008_solr, 127.0.0.1:10024_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10106_solr, 127.0.0.1:10100_solr, 127.0.0.1:10037_solr, 127.0.0.1:10109_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10103_solr, 127.0.0.1:10056_solr, 127.0.0.1:10072_solr]
collections:{.system=DocCollection(.system//clusterstate.json/307)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{"shard1":{
      "replicas":{},
      "range":"80000000-7fffffff",
      "state":"active"}}}}, last coll state: DocCollection(.system//clusterstate.json/307)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{"shard1":{
      "replicas":{},
      "range":"80000000-7fffffff",
      "state":"active"}}}
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:112)
        at org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.createSystemCollection(SimClusterStateProvider.java:1341)
        ... 39 more




Build Log:
[...truncated 13286 lines...]
   [junit4] Suite: org.apache.solr.cloud.MoveReplicaHDFSTest
   [junit4]   2> 702897 INFO  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_408E7BC14D58FA08-001/init-core-data-001
   [junit4]   2> 702898 WARN  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=22 numCloses=22
   [junit4]   2> 702898 INFO  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 702900 INFO  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 702901 INFO  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_408E7BC14D58FA08-001/tempDir-001
   [junit4]   2> 702901 INFO  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 702901 INFO  (Thread-950) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 702901 INFO  (Thread-950) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 702908 ERROR (Thread-950) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 703001 INFO  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:37063
   [junit4]   2> 703011 INFO  (zkConnectionManagerCallback-1204-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703021 INFO  (jetty-launcher-1201-thread-2) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 703021 INFO  (jetty-launcher-1201-thread-1) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 703022 INFO  (jetty-launcher-1201-thread-4) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 703022 INFO  (jetty-launcher-1201-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 703022 INFO  (jetty-launcher-1201-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 703022 INFO  (jetty-launcher-1201-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 703022 INFO  (jetty-launcher-1201-thread-3) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 703023 INFO  (jetty-launcher-1201-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6173f845{/solr,null,AVAILABLE}
   [junit4]   2> 703024 INFO  (jetty-launcher-1201-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 703024 INFO  (jetty-launcher-1201-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 703024 INFO  (jetty-launcher-1201-thread-2) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 703024 INFO  (jetty-launcher-1201-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7cd8e2a1{/solr,null,AVAILABLE}
   [junit4]   2> 703025 INFO  (jetty-launcher-1201-thread-4) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 703025 INFO  (jetty-launcher-1201-thread-4) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 703025 INFO  (jetty-launcher-1201-thread-4) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 703025 INFO  (jetty-launcher-1201-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@f76d42d{/solr,null,AVAILABLE}
   [junit4]   2> 703025 INFO  (jetty-launcher-1201-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@32967b5b{HTTP/1.1,[http/1.1]}{127.0.0.1:36856}
   [junit4]   2> 703025 INFO  (jetty-launcher-1201-thread-4) [    ] o.e.j.s.Server Started @703075ms
   [junit4]   2> 703025 INFO  (jetty-launcher-1201-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36856}
   [junit4]   2> 703026 ERROR (jetty-launcher-1201-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 703026 INFO  (jetty-launcher-1201-thread-4) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 703026 INFO  (jetty-launcher-1201-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 703026 INFO  (jetty-launcher-1201-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 703026 INFO  (jetty-launcher-1201-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 703026 INFO  (jetty-launcher-1201-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-28T17:44:41.476Z
   [junit4]   2> 703028 INFO  (jetty-launcher-1201-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 703028 INFO  (jetty-launcher-1201-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 703028 INFO  (jetty-launcher-1201-thread-3) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 703029 INFO  (jetty-launcher-1201-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@105cd30f{/solr,null,AVAILABLE}
   [junit4]   2> 703029 INFO  (jetty-launcher-1201-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@55908c2{HTTP/1.1,[http/1.1]}{127.0.0.1:40869}
   [junit4]   2> 703029 INFO  (jetty-launcher-1201-thread-2) [    ] o.e.j.s.Server Started @703078ms
   [junit4]   2> 703029 INFO  (jetty-launcher-1201-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40869}
   [junit4]   2> 703029 ERROR (jetty-launcher-1201-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 703029 INFO  (jetty-launcher-1201-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 703029 INFO  (jetty-launcher-1201-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 703029 INFO  (jetty-launcher-1201-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 703029 INFO  (jetty-launcher-1201-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 703029 INFO  (jetty-launcher-1201-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-28T17:44:41.479Z
   [junit4]   2> 703031 INFO  (zkConnectionManagerCallback-1206-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703031 INFO  (jetty-launcher-1201-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 703036 INFO  (jetty-launcher-1201-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7b95d28e{HTTP/1.1,[http/1.1]}{127.0.0.1:36558}
   [junit4]   2> 703036 INFO  (jetty-launcher-1201-thread-3) [    ] o.e.j.s.Server Started @703085ms
   [junit4]   2> 703036 INFO  (jetty-launcher-1201-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36558}
   [junit4]   2> 703036 ERROR (jetty-launcher-1201-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 703036 INFO  (jetty-launcher-1201-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 703036 INFO  (jetty-launcher-1201-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 703036 INFO  (jetty-launcher-1201-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 703036 INFO  (jetty-launcher-1201-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 703036 INFO  (jetty-launcher-1201-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-28T17:44:41.486Z
   [junit4]   2> 703038 INFO  (jetty-launcher-1201-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@49171dd1{HTTP/1.1,[http/1.1]}{127.0.0.1:37513}
   [junit4]   2> 703038 INFO  (jetty-launcher-1201-thread-1) [    ] o.e.j.s.Server Started @703088ms
   [junit4]   2> 703038 INFO  (jetty-launcher-1201-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37513}
   [junit4]   2> 703040 ERROR (jetty-launcher-1201-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 703040 INFO  (jetty-launcher-1201-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 703040 INFO  (jetty-launcher-1201-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 703040 INFO  (jetty-launcher-1201-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 703040 INFO  (jetty-launcher-1201-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 703040 INFO  (jetty-launcher-1201-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-28T17:44:41.490Z
   [junit4]   2> 703042 INFO  (zkConnectionManagerCallback-1210-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703043 INFO  (jetty-launcher-1201-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 703043 INFO  (zkConnectionManagerCallback-1208-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703043 INFO  (zkConnectionManagerCallback-1212-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703043 INFO  (jetty-launcher-1201-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 703064 INFO  (jetty-launcher-1201-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 703331 INFO  (jetty-launcher-1201-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37063/solr
   [junit4]   2> 703374 INFO  (zkConnectionManagerCallback-1216-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703375 INFO  (jetty-launcher-1201-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37063/solr
   [junit4]   2> 703384 DEBUG (jetty-launcher-1201-thread-2) [n:127.0.0.1:40869_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$43/392580282@437efedc
   [junit4]   2> 703384 INFO  (zkConnectionManagerCallback-1220-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703387 INFO  (zkConnectionManagerCallback-1222-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703389 INFO  (jetty-launcher-1201-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37063/solr
   [junit4]   2> 703390 DEBUG (jetty-launcher-1201-thread-4) [n:127.0.0.1:36856_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$43/392580282@70244d97
   [junit4]   2> 703391 INFO  (zkConnectionManagerCallback-1228-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703394 INFO  (zkConnectionManagerCallback-1226-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703395 DEBUG (jetty-launcher-1201-thread-3) [n:127.0.0.1:36558_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$43/392580282@7c8fe9ae
   [junit4]   2> 703419 INFO  (zkConnectionManagerCallback-1230-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703434 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/clusterprops.json
   [junit4]   2> 703436 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703436 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
   [junit4]   2> 703437 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 703438 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
   [junit4]   2> 703439 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/security.json
   [junit4]   2> 703443 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/clusterprops.json
   [junit4]   2> 703443 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703443 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
   [junit4]   2> 703444 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 703444 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
   [junit4]   2> 703445 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/security.json
   [junit4]   2> 703448 DEBUG (jetty-launcher-1201-thread-3) [n:127.0.0.1:36558_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/72414463843303434-127.0.0.1:36558_solr-n_0000000000
   [junit4]   2> 703448 INFO  (jetty-launcher-1201-thread-3) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36558_solr
   [junit4]   2> 703448 DEBUG (jetty-launcher-1201-thread-2) [n:127.0.0.1:40869_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/72414463843303431-127.0.0.1:40869_solr-n_0000000001
   [junit4]   2> 703449 INFO  (jetty-launcher-1201-thread-3) [n:127.0.0.1:36558_solr    ] o.a.s.c.Overseer Overseer (id=72414463843303434-127.0.0.1:36558_solr-n_0000000000) starting
   [junit4]   2> 703449 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/overseer_elect/election/72414463843303434-127.0.0.1:36558_solr-n_0000000000
   [junit4]   2> 703450 DEBUG (jetty-launcher-1201-thread-2) [n:127.0.0.1:40869_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/72414463843303434-127.0.0.1:36558_solr-n_0000000000 to know if I could be the leader
   [junit4]   2> 703451 INFO  (jetty-launcher-1201-thread-2) [n:127.0.0.1:40869_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40869_solr
   [junit4]   2> 703452 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 703452 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 703459 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703471 INFO  (zkCallback-1229-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 703475 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703476 INFO  (zkCallback-1221-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 703491 INFO  (zkConnectionManagerCallback-1241-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703493 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/clusterprops.json
   [junit4]   2> 703493 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703493 INFO  (jetty-launcher-1201-thread-3) [n:127.0.0.1:36558_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 703493 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
   [junit4]   2> 703493 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 703494 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
   [junit4]   2> 703494 INFO  (jetty-launcher-1201-thread-3) [n:127.0.0.1:36558_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37063/solr ready
   [junit4]   2> 703515 INFO  (OverseerStateUpdate-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:36558_solr
   [junit4]   2> 703515 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 703516 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue-work
   [junit4]   2> 703517 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 703517 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor Cleaning up work-queue. #Running tasks: 0
   [junit4]   2> 703517 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor RunningTasks: []
   [junit4]   2> 703517 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor BlockedTasks: []
   [junit4]   2> 703517 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor CompletedTasks: []
   [junit4]   2> 703517 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor RunningZKTasks: []
   [junit4]   2> 703517 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskQueue Peeking for top 100 elements. ExcludeSet: []
   [junit4]   2> 703517 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 703519 INFO  (zkConnectionManagerCallback-1246-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703519 DEBUG (jetty-launcher-1201-thread-3) [n:127.0.0.1:36558_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:36558_solr as DOWN
   [junit4]   2> 703520 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/clusterprops.json
   [junit4]   2> 703520 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703520 INFO  (jetty-launcher-1201-thread-2) [n:127.0.0.1:40869_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 703520 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 703520 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
   [junit4]   2> 703520 INFO  (jetty-launcher-1201-thread-3) [n:127.0.0.1:36558_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 703520 INFO  (jetty-launcher-1201-thread-3) [n:127.0.0.1:36558_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36558_solr
   [junit4]   2> 703520 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 703520 DEBUG (OverseerAutoScalingTriggerThread-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 703521 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 703521 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
   [junit4]   2> 703521 DEBUG (OverseerStateUpdate-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:36558_solr"} current state version: 0
   [junit4]   2> 703521 DEBUG (OverseerStateUpdate-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:36558_solr
   [junit4]   2> 703521 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 703521 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 703521 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 703521 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 703522 INFO  (jetty-launcher-1201-thread-2) [n:127.0.0.1:40869_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37063/solr ready
   [junit4]   2> 703522 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703522 INFO  (zkCallback-1229-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 703522 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/clusterprops.json
   [junit4]   2> 703523 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703523 INFO  (jetty-launcher-1201-thread-4) [n:127.0.0.1:36856_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 703523 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/autoscaling.json
   [junit4]   2> 703523 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
   [junit4]   2> 703523 DEBUG (OverseerAutoScalingTriggerThread-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 703523 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 703524 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
   [junit4]   2> 703524 DEBUG (OverseerAutoScalingTriggerThread-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 703524 DEBUG (OverseerAutoScalingTriggerThread-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 703524 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/security.json
   [junit4]   2> 703525 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703526 DEBUG (jetty-launcher-1201-thread-4) [n:127.0.0.1:36856_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/72414463843303432-127.0.0.1:36856_solr-n_0000000002
   [junit4]   2> 703526 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/overseer_elect/election/72414463843303431-127.0.0.1:40869_solr-n_0000000001
   [junit4]   2> 703526 DEBUG (jetty-launcher-1201-thread-4) [n:127.0.0.1:36856_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/72414463843303431-127.0.0.1:40869_solr-n_0000000001 to know if I could be the leader
   [junit4]   2> 703527 DEBUG (jetty-launcher-1201-thread-4) [n:127.0.0.1:36856_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:36856_solr as DOWN
   [junit4]   2> 703527 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 703528 INFO  (jetty-launcher-1201-thread-4) [n:127.0.0.1:36856_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 703528 INFO  (jetty-launcher-1201-thread-4) [n:127.0.0.1:36856_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36856_solr
   [junit4]   2> 703528 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 703528 DEBUG (OverseerStateUpdate-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:36856_solr"} current state version: 0
   [junit4]   2> 703528 DEBUG (OverseerStateUpdate-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:36856_solr
   [junit4]   2> 703528 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 703528 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 703528 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 703531 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703531 INFO  (zkCallback-1229-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 703543 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703543 INFO  (zkCallback-1221-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 703543 INFO  (jetty-launcher-1201-thread-2) [n:127.0.0.1:40869_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 703544 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703559 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703573 INFO  (jetty-launcher-1201-thread-2) [n:127.0.0.1:40869_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40869.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79eb9e4
   [junit4]   2> 703575 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703581 INFO  (jetty-launcher-1201-thread-2) [n:127.0.0.1:40869_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40869.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79eb9e4
   [junit4]   2> 703581 INFO  (jetty-launcher-1201-thread-2) [n:127.0.0.1:40869_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40869.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79eb9e4
   [junit4]   2> 703582 INFO  (jetty-launcher-1201-thread-2) [n:127.0.0.1:40869_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_408E7BC14D58FA08-001/tempDir-001/node1/.
   [junit4]   2> 703583 INFO  (zkCallback-1240-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 703583 INFO  (zkCallback-1245-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 703583 INFO  (zkCallback-1227-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 703584 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/autoscaling/events/.scheduled_maintenance
   [junit4]   2> 703585 DEBUG (OverseerAutoScalingTriggerThread-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:40869_solr, 127.0.0.1:36856_solr, 127.0.0.1:36558_solr]
   [junit4]   2> 703615 INFO  (jetty-launcher-1201-thread-3) [n:127.0.0.1:36558_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 703623 DEBUG (OverseerAutoScalingTriggerThread-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 703623 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 703638 INFO  (jetty-launcher-1201-thread-3) [n:127.0.0.1:36558_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36558.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79eb9e4
   [junit4]   2> 703643 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 703643 DEBUG (OverseerAutoScalingTriggerThread-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 703643 DEBUG (ScheduledTrigger-2219-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 703643 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 703646 INFO  (jetty-launcher-1201-thread-3) [n:127.0.0.1:36558_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36558.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79eb9e4
   [junit4]   2> 703646 INFO  (jetty-launcher-1201-thread-3) [n:127.0.0.1:36558_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36558.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79eb9e4
   [junit4]   2> 703647 INFO  (jetty-launcher-1201-thread-3) [n:127.0.0.1:36558_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_408E7BC14D58FA08-001/tempDir-001/node4/.
   [junit4]   2> 703648 INFO  (zkConnectionManagerCallback-1253-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703649 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/clusterprops.json
   [junit4]   2> 703649 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703649 INFO  (jetty-launcher-1201-thread-4) [n:127.0.0.1:36856_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 703649 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
   [junit4]   2> 703649 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 703649 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
   [junit4]   2> 703650 INFO  (jetty-launcher-1201-thread-4) [n:127.0.0.1:36856_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37063/solr ready
   [junit4]   2> 703657 INFO  (jetty-launcher-1201-thread-4) [n:127.0.0.1:36856_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 703673 INFO  (jetty-launcher-1201-thread-4) [n:127.0.0.1:36856_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36856.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79eb9e4
   [junit4]   2> 703680 INFO  (jetty-launcher-1201-thread-4) [n:127.0.0.1:36856_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36856.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79eb9e4
   [junit4]   2> 703681 INFO  (jetty-launcher-1201-thread-4) [n:127.0.0.1:36856_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36856.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79eb9e4
   [junit4]   2> 703681 INFO  (jetty-launcher-1201-thread-4) [n:127.0.0.1:36856_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_408E7BC14D58FA08-001/tempDir-001/node3/.
   [junit4]   2> 703712 INFO  (jetty-launcher-1201-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37063/solr
   [junit4]   2> 703724 INFO  (zkConnectionManagerCallback-1258-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703726 DEBUG (jetty-launcher-1201-thread-1) [n:127.0.0.1:37513_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$43/392580282@43e096f1
   [junit4]   2> 703727 INFO  (zkConnectionManagerCallback-1260-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703732 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/clusterprops.json
   [junit4]   2> 703732 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703733 INFO  (jetty-launcher-1201-thread-1) [n:127.0.0.1:37513_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 703733 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
   [junit4]   2> 703733 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 703733 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
   [junit4]   2> 703734 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/security.json
   [junit4]   2> 703735 DEBUG (jetty-launcher-1201-thread-1) [n:127.0.0.1:37513_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/72414463843303439-127.0.0.1:37513_solr-n_0000000003
   [junit4]   2> 703736 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/overseer_elect/election/72414463843303432-127.0.0.1:36856_solr-n_0000000002
   [junit4]   2> 703736 DEBUG (jetty-launcher-1201-thread-1) [n:127.0.0.1:37513_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/72414463843303432-127.0.0.1:36856_solr-n_0000000002 to know if I could be the leader
   [junit4]   2> 703736 DEBUG (jetty-launcher-1201-thread-1) [n:127.0.0.1:37513_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:37513_solr as DOWN
   [junit4]   2> 703737 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 703737 INFO  (jetty-launcher-1201-thread-1) [n:127.0.0.1:37513_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 703737 INFO  (jetty-launcher-1201-thread-1) [n:127.0.0.1:37513_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37513_solr
   [junit4]   2> 703737 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 703738 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 703738 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 703738 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 703738 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 703738 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 703738 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 703738 DEBUG (OverseerStateUpdate-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:37513_solr"} current state version: 0
   [junit4]   2> 703738 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 703738 DEBUG (OverseerStateUpdate-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:37513_solr
   [junit4]   2> 703739 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703739 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703739 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703739 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703739 INFO  (zkCallback-1240-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 703739 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703739 INFO  (zkCallback-1229-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 703739 INFO  (zkCallback-1221-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 703739 INFO  (zkCallback-1245-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 703739 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703739 INFO  (zkCallback-1227-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 703746 INFO  (zkCallback-1252-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 703746 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703751 INFO  (zkCallback-1259-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 703766 INFO  (zkConnectionManagerCallback-1267-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703767 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/clusterprops.json
   [junit4]   2> 703767 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703768 INFO  (jetty-launcher-1201-thread-1) [n:127.0.0.1:37513_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 703768 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
   [junit4]   2> 703768 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 703768 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
   [junit4]   2> 703770 INFO  (jetty-launcher-1201-thread-1) [n:127.0.0.1:37513_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37063/solr ready
   [junit4]   2> 703770 INFO  (jetty-launcher-1201-thread-1) [n:127.0.0.1:37513_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 703785 INFO  (jetty-launcher-1201-thread-1) [n:127.0.0.1:37513_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37513.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79eb9e4
   [junit4]   2> 703793 INFO  (jetty-launcher-1201-thread-1) [n:127.0.0.1:37513_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37513.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79eb9e4
   [junit4]   2> 703794 INFO  (jetty-launcher-1201-thread-1) [n:127.0.0.1:37513_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37513.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79eb9e4
   [junit4]   2> 703794 INFO  (jetty-launcher-1201-thread-1) [n:127.0.0.1:37513_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_408E7BC14D58FA08-001/tempDir-001/node2/.
   [junit4]   2> 703834 INFO  (zkConnectionManagerCallback-1270-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703838 INFO  (zkConnectionManagerCallback-1275-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 703839 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 703840 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/clusterprops.json
   [junit4]   2> 703840 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 703840 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 703840 INFO  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 703840 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
   [junit4]   2> 703840 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 703840 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
   [junit4]   2> 703841 INFO  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37063/solr ready
   [junit4]   2> 703845 INFO  (qtp415503312-5428) [n:127.0.0.1:36558_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 703845 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/collection-queue-work
   [junit4]   2> 703846 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/overseer/collection-queue-work/qnr-0000000000
   [junit4]   2> 703846 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 703846 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.ZkDistributedQueue Found child node with improper name: qnr-0000000000
   [junit4]   2> 703846 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/collection-queue-work
   [junit4]   2> 703846 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 703846 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.ZkDistributedQueue Found child node with improper name: qnr-0000000000
   [junit4]   2> 703847 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskQueue Returning topN elements: [/overseer/collection-queue-work/qn-0000000000, ]
   [junit4]   2> 703847 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor Got 1 tasks from work-queue : [[org.apache.solr.cloud.OverseerTaskQueue$QueueEvent@2e684ecf]]
   [junit4]   2> 703847 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor Marked task [/overseer/collection-queue-work/qn-0000000000] as running
   [junit4]   2> 703847 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{"operation":"overseerstatus"}
   [junit4]   2> 703847 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor Cleaning up work-queue. #Running tasks: 1
   [junit4]   2> 703847 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor RunningTasks: [/overseer/collection-queue-work/qn-0000000000]
   [junit4]   2> 703847 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor BlockedTasks: []
   [junit4]   2> 703847 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor CompletedTasks: []
   [junit4]   2> 703847 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor RunningZKTasks: [/overseer/collection-queue-work/qn-0000000000]
   [junit4]   2> 703848 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskQueue Peeking for top 99 elements. ExcludeSet: [/overseer/collection-queue-work/qn-0000000000],[]
   [junit4]   2> 703848 DEBUG (OverseerThreadFactory-2221-thread-1-processing-n:127.0.0.1:36558_solr) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor Runner processing /overseer/collection-queue-work/qn-0000000000
   [junit4]   2> 703848 DEBUG (OverseerThreadFactory-2221-thread-1-processing-n:127.0.0.1:36558_solr) [n:127.0.0.1:36558_solr    ] o.a.s.c.a.c.OverseerCollectionMessageHandler OverseerCollectionMessageHandler.processMessage : overseerstatus , {"operation":"overseerstatus"}
   [junit4]   2> 703856 DEBUG (OverseerThreadFactory-2221-thread-1-processing-n:127.0.0.1:36558_solr) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor Completed task:[/overseer/collection-queue-work/qn-0000000000]
   [junit4]   2> 703857 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: data: /solr/overseer/collection-queue-work/qnr-0000000000
   [junit4]   2> 703857 DEBUG (zkCallback-1229-thread-1) [    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000000 state SyncConnected latchEventType null
   [junit4]   2> 703857 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/collection-queue-work
   [junit4]   2> 703858 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 703858 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.ZkDistributedQueue Found child node with improper name: qnr-0000000000
   [junit4]   2> 703860 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/collection-queue-work
   [junit4]   2> 703860 DEBUG (OverseerThreadFactory-2221-thread-1-processing-n:127.0.0.1:36558_solr) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor Marked task [/overseer/collection-queue-work/qn-0000000000] as completed.
   [junit4]   2> 703862 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 703861 INFO  (qtp415503312-5428) [n:127.0.0.1:36558_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=16
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 703875 DEBUG (OverseerThreadFactory-2221-thread-1-processing-n:127.0.0.1:36558_solr) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor RunningTasks: []
   [junit4]   2> 703875 DEBUG (OverseerThreadFactory-2221-thread-1-processing-n:127.0.0.1:36558_solr) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor BlockedTasks: []
   [junit4]   2> 703875 DEBUG (OverseerThreadFactory-2221-thread-1-processing-n:127.0.0.1:36558_solr) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor CompletedTasks: [/overseer/collection-queue-work/qn-0000000000]
   [junit4]   2> 703875 DEBUG (OverseerThreadFactory-2221-thread-1-processing-n:127.0.0.1:36558_solr) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor RunningZKTasks: [/overseer/collection-queue-work/qn-0000000000]
   [junit4]   2> 703875 DEBUG (OverseerThreadFactory-2221-thread-1-processing-n:127.0.0.1:36558_solr) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Message id:/overseer/collection-queue-work/qn-0000000000 complete, response:{leader=127.0.0.1:36558_solr,overseer_queue_size=0,overseer_work_queue_size=0,overseer_collection_queue_size=2,overseer_operations={am_i_leader={requests=4,errors=0,avgRequestsPerSecond=11.872227584429234,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=0.28632951249505706,medianRequestTime=0.275529,75thPcRequestTime=0.275529,95thPcRequestTime=0.423896,99thPcRequestTime=0.423896,999thPcRequestTime=0.423896},downnode={requests=3,errors=0,avgRequestsPerSecond=9.075289582507759,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=0.04920866869043996,medianRequestTime=0.050399,75thPcRequestTime=0.054304,95thPcRequestTime=0.054304,99thPcRequestTime=0.054304,999thPcRequestTime=0.054304}},collection_operations={am_i_leader={requests=3,errors=0,avgRequestsPerSecond=8.920459412342316,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=0.33454169836195874,medianRequestTime=0.302221,75thPcRequestTime=0.420053,95thPcRequestTime=0.420053,99thPcRequestTime=0.420053,999thPcRequestTime=0.420053},overseerstatus={requests=0,errors=0,avgRequestsPerSecond=0.0,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=0.0,medianRequestTime=0.0,75thPcRequestTime=0.0,95thPcRequestTime=0.0,99thPcRequestTime=0.0,999thPcRequestTime=0.0}},overseer_queue={},overseer_internal_queue={peek={avgRequestsPerSecond=2.9810920569544437,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=0.228764,medianRequestTime=0.228764,75thPcRequestTime=0.228764,95thPcRequestTime=0.228764,99thPcRequestTime=0.228764,999thPcRequestTime=0.228764}},collection_queue={peektopn_wait2000={avgRequestsPerSecond=2.989718382400671,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=329.680884,medianRequestTime=329.680884,75thPcRequestTime=329.680884,95thPcRequestTime=329.680884,99thPcRequestTime=329.680884,999thPcRequestTime=329.680884}}}
   [junit4]   2> 703904 WARN  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 703913 WARN  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 703915 INFO  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 703932 INFO  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_36269_hdfs____.1micro/webapp
   [junit4]   2> 704308 INFO  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36269
   [junit4]   2> 704393 WARN  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 704399 INFO  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 704415 INFO  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_33315_datanode____7lodjy/webapp
   [junit4]   2> 704645 DEBUG (ScheduledTrigger-2219-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
   [junit4]   2> 705007 INFO  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33315
   [junit4]   2> 705275 WARN  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 705276 INFO  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 705310 INFO  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_45734_datanode____txakgu/webapp
   [junit4]   2> 705360 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_408E7BC14D58FA08-001/tempDir-002/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_408E7BC14D58FA08-001/tempDir-002/hdfsBaseDir/data/data2/]]  heartbeating to localhost/127.0.0.1:33240) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 705388 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x135890d15e22bd: from storage DS-ee077b84-a11d-4d96-b9f1-886151750dae node DatanodeRegistration(127.0.0.1:38029, datanodeUuid=8e57cba8-55ee-4c5a-8eaf-80fbcabebefe, infoPort=38208, infoSecurePort=0, ipcPort=42798, storageInfo=lv=-56;cid=testClusterID;nsid=949270109;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 705388 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x135890d15e22bd: from storage DS-2e268a09-f6d1-44d7-a6cb-d87c2584cdf3 node DatanodeRegistration(127.0.0.1:38029, datanodeUuid=8e57cba8-55ee-4c5a-8eaf-80fbcabebefe, infoPort=38208, infoSecurePort=0, ipcPort=42798, storageInfo=lv=-56;cid=testClusterID;nsid=949270109;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 705678 DEBUG (ScheduledTrigger-2219-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
   [junit4]   2> 705728 INFO  (SUITE-MoveReplicaHDFSTest-seed#[408E7BC14D58FA08]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45734
   [junit4]   2> 705820 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_408E7BC14D58FA08-001/tempDir-002/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_408E7BC14D58FA08-001/tempDir-002/hdfsBaseDir/data/data4/]]  heartbeating to localhost/127.0.0.1:33240) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 705832 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x135890ebd63cd9: from storage DS-5430302c-6fd7-474c-aef2-c39ebf5caf09 node DatanodeRegistration(127.0.0.1:38237, datanodeUuid=60f281a9-25fc-4d63-9271-77206a214109, infoPort=33469, infoSecurePort=0, ipcPort=36715, storageInfo=lv=-56;cid=testClusterID;nsid=949270109;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 705832 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x135890ebd63cd9: from storage DS-595c19d1-b351-4af5-afea-63203350850d node DatanodeRegistration(127.0.0.1:38237, datanodeUuid=60f281a9-25fc-4d63-9271-77206a214109, infoPort=33469, infoSecurePort=0, ipcPort=36715, storageInfo=lv=-56;cid=testClusterID;nsid=949270109;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 705850 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor Got 0 tasks from work-queue : [[]]
   [junit4]   2> 705850 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor Cleaning up work-queue. #Running tasks: 0
   [junit4]   2> 705850 INFO  (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 705851 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor RunningTasks: []
   [junit4]   2> 705851 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor BlockedTasks: []
   [junit4]   2> 705851 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor CompletedTasks: []
   [junit4]   2> 705851 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskProcessor RunningZKTasks: []
   [junit4]   2> 705851 DEBUG (OverseerCollectionConfigSetProcessor-72414463843303434-127.0.0.1:36558_solr-n_0000000000) [n:127.0.0.1:36558_solr    ] o.a.s.c.OverseerTaskQueue Peeking for top 100 elements. ExcludeSet: []
   [junit4]   2> 705978 INFO  (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[408E7BC14D58FA08]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNormalMove
   [junit4]   2> 705979 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/clusterprops.json
   [junit4]   2> 705979 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 705979 INFO  (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[408E7BC14D58FA08]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 705980 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
   [junit4]   2> 705980 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 705980 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
   [junit4]   2> 705981 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: data: /solr/aliases.json
   [junit4]   2> 705981 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: data: /solr/aliases.json
   [junit4]   2> 705981 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: data: /solr/aliases.json
   [junit4]   2> 705981 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: data: /solr

[...truncated too long message...]

onfigure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null909221363
     [copy] Copying 238 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null909221363
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null909221363

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 63ms :: artifacts dl 55ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 314 minutes 10 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-NightlyTests-master - Build # 31 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-master/31/

4 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:45903/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:45903/collection1
        at __randomizedtesting.SeedInfo.seed([EDC58AE6A1D392EC:6591B53C0F2FFF14]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:484)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:463)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:1590)
        at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:215)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 53 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest:     1) Thread[id=2468, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)    2) Thread[id=2469, name=zkConnectionManagerCallback-706-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=2466, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=2467, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]-SendThread(127.0.0.1:44069), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest:
   1) Thread[id=2468, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
   2) Thread[id=2469, name=zkConnectionManagerCallback-706-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=2466, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=2467, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]-SendThread(127.0.0.1:44069), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)
        at __randomizedtesting.SeedInfo.seed([EDC58AE6A1D392EC]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=2469, name=zkConnectionManagerCallback-706-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=2467, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]-SendThread(127.0.0.1:44069), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=2469, name=zkConnectionManagerCallback-706-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=2467, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]-SendThread(127.0.0.1:44069), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
        at __randomizedtesting.SeedInfo.seed([EDC58AE6A1D392EC]:0)


FAILED:  org.apache.solr.cloud.hdfs.HdfsBasicDistributedZk2Test.test

Error Message:
.response[id=100][rnd_b]==null

Stack Trace:
junit.framework.AssertionFailedError: .response[id=100][rnd_b]==null
        at __randomizedtesting.SeedInfo.seed([EDC58AE6A1D392EC:6591B53C0F2FFF14]:0)
        at junit.framework.Assert.fail(Assert.java:50)
        at org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:930)
        at org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:613)
        at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:589)
        at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:568)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkQueries(AbstractFullDistribZkTestBase.java:1058)
        at org.apache.solr.cloud.BasicDistributedZk2Test.test(BasicDistributedZk2Test.java:102)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13374 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/init-core-data-001
   [junit4]   2> 411545 INFO  (SUITE-ChaosMonkeyNothingIsSafeTest-seed#[EDC58AE6A1D392EC]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 411558 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 411579 INFO  (Thread-386) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 411579 INFO  (Thread-386) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 411599 ERROR (Thread-386) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 411678 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.ZkTestServer start zk server on port:44069
   [junit4]   2> 411709 INFO  (zkConnectionManagerCallback-678-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 411726 INFO  (zkConnectionManagerCallback-680-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 411730 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 411731 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 411732 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 411733 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 411734 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 411735 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 411735 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 411736 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 411737 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 411738 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 411739 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 411740 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use TLOG replicas unless explicitly asked otherwise
   [junit4]   2> 411916 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 411931 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 411931 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 411932 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 411932 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1e2b1c56{/,null,AVAILABLE}
   [junit4]   2> 411947 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7526fc0c{HTTP/1.1,[http/1.1]}{127.0.0.1:42076}
   [junit4]   2> 411947 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.Server Started @412151ms
   [junit4]   2> 411947 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=42076, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/control-001/cores}
   [junit4]   2> 411947 ERROR (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 411947 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 411947 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 411947 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 411947 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 411947 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-05T17:35:44.120Z
   [junit4]   2> 411963 INFO  (zkConnectionManagerCallback-682-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 411964 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 411964 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/control-001/solr.xml
   [junit4]   2> 411969 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 411969 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 411977 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5f7b50f5, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 412392 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44069/solr
   [junit4]   2> 412394 INFO  (zkConnectionManagerCallback-686-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 412396 INFO  (zkConnectionManagerCallback-688-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 412513 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [n:127.0.0.1:42076_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42076_
   [junit4]   2> 412514 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [n:127.0.0.1:42076_    ] o.a.s.c.Overseer Overseer (id=73440295155793924-127.0.0.1:42076_-n_0000000000) starting
   [junit4]   2> 412520 INFO  (zkConnectionManagerCallback-695-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 412522 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [n:127.0.0.1:42076_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44069/solr ready
   [junit4]   2> 412523 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [n:127.0.0.1:42076_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42076_
   [junit4]   2> 412530 INFO  (OverseerStateUpdate-73440295155793924-127.0.0.1:42076_-n_0000000000) [n:127.0.0.1:42076_    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:42076_
   [junit4]   2> 412548 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [n:127.0.0.1:42076_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 412550 INFO  (zkCallback-694-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 412550 INFO  (OverseerStateUpdate-73440295155793924-127.0.0.1:42076_-n_0000000000) [n:127.0.0.1:42076_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 412564 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [n:127.0.0.1:42076_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f7b50f5
   [junit4]   2> 412571 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [n:127.0.0.1:42076_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f7b50f5
   [junit4]   2> 412572 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [n:127.0.0.1:42076_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f7b50f5
   [junit4]   2> 412573 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [n:127.0.0.1:42076_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/control-001/cores
   [junit4]   2> 412592 INFO  (zkConnectionManagerCallback-701-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 412593 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 412595 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44069/solr ready
   [junit4]   2> 412596 INFO  (qtp1351479358-2422) [n:127.0.0.1:42076_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:42076_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 412599 INFO  (OverseerThreadFactory-1081-thread-1-processing-n:127.0.0.1:42076_) [n:127.0.0.1:42076_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 412709 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 412710 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 413747 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 413791 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 413885 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 413905 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 413906 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f7b50f5
   [junit4]   2> 413906 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 413909 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=72.845703125, floorSegmentMB=1.892578125, forceMergeDeletesPctAllowed=4.454635158640582, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=35.722003686897935
   [junit4]   2> 413923 WARN  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 413962 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 413962 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 413963 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
   [junit4]   2> 413963 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 413964 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=50, maxMergedSegmentMB=33.296875, floorSegmentMB=1.87109375, forceMergeDeletesPctAllowed=13.843292993747426, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.10313148112635276, deletesPctAllowed=27.29299180116365
   [junit4]   2> 413964 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@243546f8[control_collection_shard1_replica_n1] main]
   [junit4]   2> 413966 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 413966 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 413966 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 413967 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613507797859696640
   [junit4]   2> 413971 INFO  (searcherExecutor-1086-thread-1-processing-n:127.0.0.1:42076_ x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@243546f8[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 413979 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 413992 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 413992 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 413992 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:42076/control_collection_shard1_replica_n1/
   [junit4]   2> 413993 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 413993 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:42076/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 413993 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 413997 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:42076/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 414106 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 414110 INFO  (qtp1351479358-2417) [n:127.0.0.1:42076_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1400
   [junit4]   2> 414115 INFO  (qtp1351479358-2422) [n:127.0.0.1:42076_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 414209 INFO  (zkCallback-687-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 414601 INFO  (OverseerCollectionConfigSetProcessor-73440295155793924-127.0.0.1:42076_-n_0000000000) [n:127.0.0.1:42076_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 415116 INFO  (qtp1351479358-2422) [n:127.0.0.1:42076_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:42076_&wt=javabin&version=2} status=0 QTime=2519
   [junit4]   2> 415160 INFO  (zkConnectionManagerCallback-706-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 415161 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 415162 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44069/solr ready
   [junit4]   2> 415162 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 415164 INFO  (qtp1351479358-2418) [n:127.0.0.1:42076_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=4&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 415167 INFO  (OverseerThreadFactory-1081-thread-2-processing-n:127.0.0.1:42076_) [n:127.0.0.1:42076_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 415371 WARN  (OverseerThreadFactory-1081-thread-2-processing-n:127.0.0.1:42076_) [n:127.0.0.1:42076_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 415374 INFO  (qtp1351479358-2418) [n:127.0.0.1:42076_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 415374 INFO  (qtp1351479358-2418) [n:127.0.0.1:42076_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=4&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=209
   [junit4]   2> 415578 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/shard-1-001 of type TLOG
   [junit4]   2> 415578 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 415590 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 415590 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 415590 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 415590 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2940909a{/,null,AVAILABLE}
   [junit4]   2> 415591 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7a18bde3{HTTP/1.1,[http/1.1]}{127.0.0.1:41115}
   [junit4]   2> 415591 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.Server Started @415796ms
   [junit4]   2> 415591 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/tempDir-001/jetty1, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=41115, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/shard-1-001/cores}
   [junit4]   2> 415592 ERROR (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 415592 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 415592 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 415592 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 415592 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 415592 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-05T17:35:47.765Z
   [junit4]   2> 415594 INFO  (zkConnectionManagerCallback-708-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 415598 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 415598 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/shard-1-001/solr.xml
   [junit4]   2> 415602 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 415602 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 415603 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5f7b50f5, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 415657 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44069/solr
   [junit4]   2> 415670 INFO  (zkConnectionManagerCallback-712-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 415690 INFO  (zkConnectionManagerCallback-714-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 415698 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 415706 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 415709 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41115_
   [junit4]   2> 415713 INFO  (zkCallback-705-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 415714 INFO  (zkCallback-687-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 415718 INFO  (zkCallback-694-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 415748 INFO  (zkCallback-713-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 415770 INFO  (zkConnectionManagerCallback-721-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 415771 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 415772 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44069/solr ready
   [junit4]   2> 415790 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 415810 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f7b50f5
   [junit4]   2> 415830 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f7b50f5
   [junit4]   2> 415830 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f7b50f5
   [junit4]   2> 415831 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/shard-1-001/cores
   [junit4]   2> 415910 INFO  (qtp1883800410-2477) [n:127.0.0.1:41115_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:41115_&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 415913 INFO  (OverseerCollectionConfigSetProcessor-73440295155793924-127.0.0.1:42076_-n_0000000000) [n:127.0.0.1:42076_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 415914 INFO  (OverseerThreadFactory-1081-thread-3-processing-n:127.0.0.1:42076_) [n:127.0.0.1:42076_ c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:41115_ for creating new replica of shard shard2 for collection collection1
   [junit4]   2> 415921 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_    x:collection1_shard2_replica_t81] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t81&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 416959 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 416983 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.s.IndexSchema [collection1_shard2_replica_t81] Schema name=test
   [junit4]   2> 417187 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 417203 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_t81' using configuration from collection collection1, trusted=true
   [junit4]   2> 417203 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_t81' (registry 'solr.core.collection1.shard2.replica_t81') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f7b50f5
   [junit4]   2> 417203 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.c.SolrCore [[collection1_shard2_replica_t81] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/shard-1-001/cores/collection1_shard2_replica_t81], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/shard-1-001/cores/collection1_shard2_replica_t81/data/]
   [junit4]   2> 417216 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=72.845703125, floorSegmentMB=1.892578125, forceMergeDeletesPctAllowed=4.454635158640582, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=35.722003686897935
   [junit4]   2> 417231 WARN  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 417314 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 417314 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 417315 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
   [junit4]   2> 417315 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 417316 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=50, maxMergedSegmentMB=33.296875, floorSegmentMB=1.87109375, forceMergeDeletesPctAllowed=13.843292993747426, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.10313148112635276, deletesPctAllowed=27.29299180116365
   [junit4]   2> 417317 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.s.SolrIndexSearcher Opening [Searcher@369f2234[collection1_shard2_replica_t81] main]
   [junit4]   2> 417318 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 417318 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 417319 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 417320 INFO  (searcherExecutor-1100-thread-1-processing-n:127.0.0.1:41115_ x:collection1_shard2_replica_t81 c:collection1 s:shard2) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.c.SolrCore [collection1_shard2_replica_t81] Registered new searcher Searcher@369f2234[collection1_shard2_replica_t81] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 417322 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613507801377669120
   [junit4]   2> 417333 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node82=0}, version=0}
   [junit4]   2> 417339 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 417339 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 417339 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:41115/collection1_shard2_replica_t81/
   [junit4]   2> 417339 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 417340 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.c.SyncStrategy http://127.0.0.1:41115/collection1_shard2_replica_t81/ has no replicas
   [junit4]   2> 417340 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 417340 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.c.ZkController collection1_shard2_replica_t81 stopping background replication from leader
   [junit4]   2> 417343 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:41115/collection1_shard2_replica_t81/ shard2
   [junit4]   2> 417494 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 417496 INFO  (qtp1883800410-2472) [n:127.0.0.1:41115_ c:collection1 s:shard2  x:collection1_shard2_replica_t81] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t81&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1576
   [junit4]   2> 417505 INFO  (qtp1883800410-2477) [n:127.0.0.1:41115_ c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:41115_&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2} status=0 QTime=1594
   [junit4]   2> 417626 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/shard-2-001 of type TLOG
   [junit4]   2> 417626 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 417627 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 417627 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 417627 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 417628 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5596c4ce{/,null,AVAILABLE}
   [junit4]   2> 417628 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5a9071c2{HTTP/1.1,[http/1.1]}{127.0.0.1:40230}
   [junit4]   2> 417628 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.Server Started @417832ms
   [junit4]   2> 417628 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/tempDir-001/jetty2, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=40230, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/shard-2-001/cores}
   [junit4]   2> 417629 ERROR (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 417630 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 417630 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 417630 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 417630 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 417630 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-05T17:35:49.803Z
   [junit4]   2> 417635 INFO  (zkConnectionManagerCallback-724-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 417636 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 417636 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/shard-2-001/solr.xml
   [junit4]   2> 417641 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 417641 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 417642 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5f7b50f5, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 417771 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44069/solr
   [junit4]   2> 417772 INFO  (zkConnectionManagerCallback-728-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 417775 INFO  (zkConnectionManagerCallback-730-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 417781 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 417785 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 417785 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40230_
   [junit4]   2> 417786 INFO  (zkCallback-694-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 417786 INFO  (zkCallback-687-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 417786 INFO  (zkCallback-705-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 417786 INFO  (zkCallback-720-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 417786 INFO  (zkCallback-713-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 417790 INFO  (zkCallback-729-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 417806 INFO  (zkConnectionManagerCallback-737-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 417807 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 417808 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44069/solr ready
   [junit4]   2> 417808 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 417824 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f7b50f5
   [junit4]   2> 417833 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f7b50f5
   [junit4]   2> 417833 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f7b50f5
   [junit4]   2> 417834 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/shard-2-001/cores
   [junit4]   2> 417869 INFO  (qtp695104570-2514) [n:127.0.0.1:40230_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:40230_&action=ADDREPLICA&collection=collection1&shard=shard3&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 417872 INFO  (OverseerCollectionConfigSetProcessor-73440295155793924-127.0.0.1:42076_-n_0000000000) [n:127.0.0.1:42076_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 417873 INFO  (OverseerThreadFactory-1081-thread-4-processing-n:127.0.0.1:42076_) [n:127.0.0.1:42076_ c:collection1 s:shard3  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:40230_ for creating new replica of shard shard3 for collection collection1
   [junit4]   2> 417877 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_    x:collection1_shard3_replica_t83] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard3_replica_t83&action=CREATE&collection=collection1&shard=shard3&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 418905 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 418943 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.s.IndexSchema [collection1_shard3_replica_t83] Schema name=test
   [junit4]   2> 419069 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 419089 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard3_replica_t83' using configuration from collection collection1, trusted=true
   [junit4]   2> 419089 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard3.replica_t83' (registry 'solr.core.collection1.shard3.replica_t83') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f7b50f5
   [junit4]   2> 419089 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.c.SolrCore [[collection1_shard3_replica_t83] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/shard-2-001/cores/collection1_shard3_replica_t83], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/shard-2-001/cores/collection1_shard3_replica_t83/data/]
   [junit4]   2> 419092 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=72.845703125, floorSegmentMB=1.892578125, forceMergeDeletesPctAllowed=4.454635158640582, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=35.722003686897935
   [junit4]   2> 419127 WARN  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 419241 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 419241 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 419242 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
   [junit4]   2> 419242 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 419243 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=50, maxMergedSegmentMB=33.296875, floorSegmentMB=1.87109375, forceMergeDeletesPctAllowed=13.843292993747426, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.10313148112635276, deletesPctAllowed=27.29299180116365
   [junit4]   2> 419244 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.s.SolrIndexSearcher Opening [Searcher@60693b08[collection1_shard3_replica_t83] main]
   [junit4]   2> 419245 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 419245 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 419246 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 419246 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613507803395129344
   [junit4]   2> 419251 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard3 to Terms{values={core_node84=0}, version=0}
   [junit4]   2> 419253 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 419253 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 419253 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40230/collection1_shard3_replica_t83/
   [junit4]   2> 419253 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 419253 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.c.SyncStrategy http://127.0.0.1:40230/collection1_shard3_replica_t83/ has no replicas
   [junit4]   2> 419253 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 419253 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.c.ZkController collection1_shard3_replica_t83 stopping background replication from leader
   [junit4]   2> 419255 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40230/collection1_shard3_replica_t83/ shard3
   [junit4]   2> 419253 INFO  (searcherExecutor-1114-thread-1-processing-n:127.0.0.1:40230_ x:collection1_shard3_replica_t83 c:collection1 s:shard3) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.c.SolrCore [collection1_shard3_replica_t83] Registered new searcher Searcher@60693b08[collection1_shard3_replica_t83] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 419407 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 419409 INFO  (qtp695104570-2509) [n:127.0.0.1:40230_ c:collection1 s:shard3  x:collection1_shard3_replica_t83] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard3_replica_t83&action=CREATE&collection=collection1&shard=shard3&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1533
   [junit4]   2> 419415 INFO  (qtp695104570-2514) [n:127.0.0.1:40230_ c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:40230_&action=ADDREPLICA&collection=collection1&shard=shard3&type=TLOG&wt=javabin&version=2} status=0 QTime=1545
   [junit4]   2> 419560 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/shard-3-001 of type TLOG
   [junit4]   2> 419561 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 419562 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 419562 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 419562 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 419563 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@412a4685{/,null,AVAILABLE}
   [junit4]   2> 419565 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@659e4a6c{HTTP/1.1,[http/1.1]}{127.0.0.1:46802}
   [junit4]   2> 419565 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.e.j.s.Server Started @419770ms
   [junit4]   2> 419565 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/tempDir-001/jetty3, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=46802, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/shard-3-001/cores}
   [junit4]   2> 419566 ERROR (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 419566 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 419566 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 419566 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 419566 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 419566 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-05T17:35:51.739Z
   [junit4]   2> 419568 INFO  (zkConnectionManagerCallback-740-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 419569 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 419569 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_EDC58AE6A1D392EC-001/shard-3-001/solr.xml
   [junit4]   2> 419573 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 419573 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 419574 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5f7b50f5, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 419773 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44069/solr
   [junit4]   2> 419775 INFO  (zkConnectionManagerCallback-744-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 419777 INFO  (zkConnectionManagerCallback-746-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 419786 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 419790 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 419790 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46802_
   [junit4]   2> 419791 INFO  (zkCallback-736-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 419791 INFO  (zkCallback-729-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 419791 INFO  (zkCallback-705-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 419802 INFO  (zkCallback-713-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 419802 INFO  (zkCallback-694-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 419802 INFO  (zkCallback-720-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 419802 INFO  (zkCallback-687-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 419803 INFO  (zkCallback-745-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 419809 INFO  (zkConnectionManagerCallback-753-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 419810 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 419811 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44069/solr ready
   [junit4]   2> 419812 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[EDC58AE6A1D392EC]) [    ] o.a.s.h.a.Metrics

[...truncated too long message...]

ing settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null560030463
     [copy] Copying 238 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null560030463
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null560030463

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 158ms :: artifacts dl 31ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 492 minutes 6 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-NightlyTests-master - Build # 32 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-master/32/

6 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsRestartWhileUpdatingTest

Error Message:
7 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsRestartWhileUpdatingTest:     1) Thread[id=170791, name=searcherExecutor-4589-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=170754, name=searcherExecutor-4575-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=170985, name=searcherExecutor-4651-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=170950, name=searcherExecutor-4637-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=170712, name=searcherExecutor-4561-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=170915, name=searcherExecutor-4623-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=170837, name=searcherExecutor-4603-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 7 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsRestartWhileUpdatingTest:
   1) Thread[id=170791, name=searcherExecutor-4589-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=170754, name=searcherExecutor-4575-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=170985, name=searcherExecutor-4651-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=170950, name=searcherExecutor-4637-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=170712, name=searcherExecutor-4561-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=170915, name=searcherExecutor-4623-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=170837, name=searcherExecutor-4603-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([6CBEDD2BE0D3C029]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsRestartWhileUpdatingTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=170791, name=searcherExecutor-4589-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=170754, name=searcherExecutor-4575-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=170985, name=searcherExecutor-4651-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=170950, name=searcherExecutor-4637-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=170712, name=searcherExecutor-4561-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=170915, name=searcherExecutor-4623-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=170837, name=searcherExecutor-4603-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=170791, name=searcherExecutor-4589-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=170754, name=searcherExecutor-4575-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=170985, name=searcherExecutor-4651-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=170950, name=searcherExecutor-4637-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=170712, name=searcherExecutor-4561-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=170915, name=searcherExecutor-4623-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=170837, name=searcherExecutor-4603-thread-1, state=WAITING, group=TGRP-HdfsRestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([6CBEDD2BE0D3C029]:0)


FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:36522/i_/mh/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:36522/i_/mh/collection1
        at __randomizedtesting.SeedInfo.seed([6CBEDD2BE0D3C029:E4EAE2F14E2FADD1]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:484)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:463)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:1590)
        at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:215)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 53 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest:     1) Thread[id=33302, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)    2) Thread[id=33300, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=33301, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]-SendThread(127.0.0.1:39955), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)    4) Thread[id=33303, name=zkConnectionManagerCallback-9312-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest:
   1) Thread[id=33302, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
   2) Thread[id=33300, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=33301, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]-SendThread(127.0.0.1:39955), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
   4) Thread[id=33303, name=zkConnectionManagerCallback-9312-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([6CBEDD2BE0D3C029]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=33301, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]-SendThread(127.0.0.1:39955), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)    2) Thread[id=33303, name=zkConnectionManagerCallback-9312-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=33301, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]-SendThread(127.0.0.1:39955), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)
   2) Thread[id=33303, name=zkConnectionManagerCallback-9312-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([6CBEDD2BE0D3C029]:0)


FAILED:  org.apache.solr.cloud.hdfs.HdfsRestartWhileUpdatingTest.test

Error Message:
There are still nodes recoverying - waited for 320 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 320 seconds
        at __randomizedtesting.SeedInfo.seed([6CBEDD2BE0D3C029:E4EAE2F14E2FADD1]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:185)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:920)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1477)
        at org.apache.solr.cloud.RestartWhileUpdatingTest.test(RestartWhileUpdatingTest.java:145)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14100 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_6CBEDD2BE0D3C029-001/init-core-data-001
   [junit4]   2> 3086397 WARN  (SUITE-ChaosMonkeyNothingIsSafeTest-seed#[6CBEDD2BE0D3C029]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=11 numCloses=11
   [junit4]   2> 3086397 INFO  (SUITE-ChaosMonkeyNothingIsSafeTest-seed#[6CBEDD2BE0D3C029]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 3086400 INFO  (SUITE-ChaosMonkeyNothingIsSafeTest-seed#[6CBEDD2BE0D3C029]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 3086400 INFO  (SUITE-ChaosMonkeyNothingIsSafeTest-seed#[6CBEDD2BE0D3C029]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 3086400 INFO  (SUITE-ChaosMonkeyNothingIsSafeTest-seed#[6CBEDD2BE0D3C029]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /i_/mh
   [junit4]   2> 3086404 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3086404 INFO  (Thread-10176) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3086404 INFO  (Thread-10176) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3086406 ERROR (Thread-10176) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3086504 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.ZkTestServer start zk server on port:39955
   [junit4]   2> 3086507 INFO  (zkConnectionManagerCallback-9284-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3086511 INFO  (zkConnectionManagerCallback-9286-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3086518 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3086520 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 3086522 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3086524 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3086525 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3086527 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3086529 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3086530 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3086532 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3086534 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3086536 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3086538 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use TLOG replicas unless explicitly asked otherwise
   [junit4]   2> 3086708 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 3086717 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3086717 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3086717 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 3086720 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4f97b529{/i_/mh,null,AVAILABLE}
   [junit4]   2> 3086720 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@28f1e17b{HTTP/1.1,[http/1.1]}{127.0.0.1:38657}
   [junit4]   2> 3086720 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.e.j.s.Server Started @3086785ms
   [junit4]   2> 3086720 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_6CBEDD2BE0D3C029-001/tempDir-001/control/data, hostContext=/i_/mh, hostPort=38657, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_6CBEDD2BE0D3C029-001/control-001/cores}
   [junit4]   2> 3086721 ERROR (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3086721 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3086721 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 3086721 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3086721 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3086721 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-12T16:12:13.123Z
   [junit4]   2> 3086729 INFO  (zkConnectionManagerCallback-9288-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3086731 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3086731 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_6CBEDD2BE0D3C029-001/control-001/solr.xml
   [junit4]   2> 3086736 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 3086737 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 3086738 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@68aba492, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3087207 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39955/solr
   [junit4]   2> 3087209 INFO  (zkConnectionManagerCallback-9292-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3087212 INFO  (zkConnectionManagerCallback-9294-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3087357 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:38657_i_%2Fmh
   [junit4]   2> 3087358 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.Overseer Overseer (id=72090068518436868-127.0.0.1:38657_i_%2Fmh-n_0000000000) starting
   [junit4]   2> 3087371 INFO  (zkConnectionManagerCallback-9301-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3087379 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:39955/solr ready
   [junit4]   2> 3087380 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38657_i_%2Fmh
   [junit4]   2> 3087380 INFO  (OverseerStateUpdate-72090068518436868-127.0.0.1:38657_i_%2Fmh-n_0000000000) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:38657_i_%2Fmh
   [junit4]   2> 3087388 INFO  (zkCallback-9300-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3087390 INFO  (OverseerStateUpdate-72090068518436868-127.0.0.1:38657_i_%2Fmh-n_0000000000) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3087395 DEBUG (OverseerAutoScalingTriggerThread-72090068518436868-127.0.0.1:38657_i_%2Fmh-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:38657_i_%2Fmh]
   [junit4]   2> 3087397 DEBUG (ScheduledTrigger-10841-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 3087407 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3087436 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68aba492
   [junit4]   2> 3087451 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68aba492
   [junit4]   2> 3087451 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68aba492
   [junit4]   2> 3087453 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_6CBEDD2BE0D3C029-001/control-001/cores
   [junit4]   2> 3087493 INFO  (zkConnectionManagerCallback-9307-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3087494 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3087497 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:39955/solr ready
   [junit4]   2> 3087499 INFO  (qtp1400687775-33256) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:38657_i_%252Fmh&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3087507 INFO  (OverseerThreadFactory-10843-thread-1-processing-n:127.0.0.1:38657_i_%2Fmh) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 3087511 DEBUG (OverseerStateUpdate-72090068518436868-127.0.0.1:38657_i_%2Fmh-n_0000000000) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "name":"control_collection",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "collection.configName":"conf1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "createNodeSet":"127.0.0.1:38657_i_%2Fmh",
   [junit4]   2>   "nrtReplicas":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 3087615 INFO  (qtp1400687775-33251) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3087620 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3087620 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3087623 DEBUG (OverseerStateUpdate-72090068518436868-127.0.0.1:38657_i_%2Fmh-n_0000000000) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"control_collection_shard1_replica_n1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:38657/i_/mh",
   [junit4]   2>   "node_name":"127.0.0.1:38657_i_%2Fmh",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 3088399 DEBUG (ScheduledTrigger-10841-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 3088646 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 3088662 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 3088758 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 3088778 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 3088778 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68aba492
   [junit4]   2> 3088779 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_6CBEDD2BE0D3C029-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_6CBEDD2BE0D3C029-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 3088783 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=81.166015625, floorSegmentMB=1.3681640625, forceMergeDeletesPctAllowed=14.083262919000399, segmentsPerTier=44.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0, deletesPctAllowed=41.514864962384394
   [junit4]   2> 3088792 WARN  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3088854 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3088854 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3088855 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
   [junit4]   2> 3088855 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3088857 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@61d62f51
   [junit4]   2> 3088857 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@79e31d8e[control_collection_shard1_replica_n1] main]
   [junit4]   2> 3088859 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3088859 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3088860 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3088860 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1614136722338086912
   [junit4]   2> 3088861 INFO  (searcherExecutor-10848-thread-1-processing-n:127.0.0.1:38657_i_%2Fmh x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@79e31d8e[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3088868 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 3088872 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3088872 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3088872 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:38657/i_/mh/control_collection_shard1_replica_n1/
   [junit4]   2> 3088873 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3088873 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:38657/i_/mh/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 3088877 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:38657/i_/mh/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 3088878 DEBUG (OverseerStateUpdate-72090068518436868-127.0.0.1:38657_i_%2Fmh-n_0000000000) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:38657/i_/mh",
   [junit4]   2>   "core":"control_collection_shard1_replica_n1",
   [junit4]   2>   "state":"active"} current state version: 0
   [junit4]   2> 3088981 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3088983 INFO  (qtp1400687775-33252) [n:127.0.0.1:38657_i_%2Fmh c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1363
   [junit4]   2> 3088983 DEBUG (OverseerStateUpdate-72090068518436868-127.0.0.1:38657_i_%2Fmh-n_0000000000) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"control_collection_shard1_replica_n1",
   [junit4]   2>   "core_node_name":"core_node2",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:38657/i_/mh",
   [junit4]   2>   "node_name":"127.0.0.1:38657_i_%2Fmh",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 3088986 INFO  (qtp1400687775-33256) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 3089085 INFO  (zkCallback-9293-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3089399 DEBUG (ScheduledTrigger-10841-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 3089509 INFO  (OverseerCollectionConfigSetProcessor-72090068518436868-127.0.0.1:38657_i_%2Fmh-n_0000000000) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3089986 INFO  (qtp1400687775-33256) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:38657_i_%252Fmh&wt=javabin&version=2} status=0 QTime=2487
   [junit4]   2> 3089994 INFO  (zkConnectionManagerCallback-9312-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3089995 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3089996 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:39955/solr ready
   [junit4]   2> 3089996 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3089998 INFO  (qtp1400687775-33251) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3090001 INFO  (OverseerThreadFactory-10843-thread-2-processing-n:127.0.0.1:38657_i_%2Fmh) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 3090005 DEBUG (OverseerStateUpdate-72090068518436868-127.0.0.1:38657_i_%2Fmh-n_0000000000) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "collection.configName":"conf1",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "createNodeSet":"",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "nrtReplicas":"1",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 3090207 WARN  (OverseerThreadFactory-10843-thread-2-processing-n:127.0.0.1:38657_i_%2Fmh) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 3090209 INFO  (qtp1400687775-33251) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 3090210 INFO  (qtp1400687775-33251) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=212
   [junit4]   2> 3090399 DEBUG (ScheduledTrigger-10841-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 3090399 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_6CBEDD2BE0D3C029-001/shard-1-001 of type TLOG
   [junit4]   2> 3090400 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 3090401 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3090401 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3090401 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 3090402 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@49cb1e36{/i_/mh,null,AVAILABLE}
   [junit4]   2> 3090402 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7b5af347{HTTP/1.1,[http/1.1]}{127.0.0.1:41041}
   [junit4]   2> 3090402 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.e.j.s.Server Started @3090468ms
   [junit4]   2> 3090402 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_6CBEDD2BE0D3C029-001/tempDir-001/jetty1, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/i_/mh, hostPort=41041, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_6CBEDD2BE0D3C029-001/shard-1-001/cores}
   [junit4]   2> 3090403 ERROR (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3090403 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3090403 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 3090403 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3090403 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3090403 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-12T16:12:16.805Z
   [junit4]   2> 3090405 INFO  (zkConnectionManagerCallback-9314-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3090406 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3090406 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_6CBEDD2BE0D3C029-001/shard-1-001/solr.xml
   [junit4]   2> 3090411 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 3090411 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 3090413 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@68aba492, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3090741 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39955/solr
   [junit4]   2> 3090744 INFO  (zkConnectionManagerCallback-9318-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3090747 INFO  (zkConnectionManagerCallback-9320-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3090758 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3090764 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3090764 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41041_i_%2Fmh
   [junit4]   2> 3090765 DEBUG (OverseerStateUpdate-72090068518436868-127.0.0.1:38657_i_%2Fmh-n_0000000000) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:41041_i_%2Fmh"} current state version: 0
   [junit4]   2> 3090766 INFO  (zkCallback-9300-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3090766 INFO  (zkCallback-9293-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3090766 INFO  (zkCallback-9319-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3090767 INFO  (zkCallback-9311-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3090795 INFO  (zkConnectionManagerCallback-9327-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3090796 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3090797 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:39955/solr ready
   [junit4]   2> 3090798 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3090830 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68aba492
   [junit4]   2> 3090852 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68aba492
   [junit4]   2> 3090852 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68aba492
   [junit4]   2> 3090854 INFO  (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[6CBEDD2BE0D3C029]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_6CBEDD2BE0D3C029-001/shard-1-001/cores
   [junit4]   2> 3090896 INFO  (qtp1400687775-33257) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:41041_i_%252Fmh&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3090899 INFO  (OverseerCollectionConfigSetProcessor-72090068518436868-127.0.0.1:38657_i_%2Fmh-n_0000000000) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3090904 INFO  (qtp1400687775-33255) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=1
   [junit4]   2> 3090908 INFO  (qtp1400687775-33256) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3090909 INFO  (qtp1878175356-33310) [n:127.0.0.1:41041_i_%2Fmh    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3090910 INFO  (OverseerThreadFactory-10843-thread-3-processing-n:127.0.0.1:38657_i_%2Fmh) [n:127.0.0.1:38657_i_%2Fmh c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:41041_i_%2Fmh for creating new replica of shard shard2 for collection collection1
   [junit4]   2> 3090914 INFO  (qtp1878175356-33305) [n:127.0.0.1:41041_i_%2Fmh    x:collection1_shard2_replica_t41] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 3090917 DEBUG (OverseerStateUpdate-72090068518436868-127.0.0.1:38657_i_%2Fmh-n_0000000000) [n:127.0.0.1:38657_i_%2Fmh    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1_shard2_replica_t41",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:41041/i_/mh",
   [junit4]   2>   "node_name":"127.0.0.1:41041_i_%2Fmh",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 3091400 DEBUG (ScheduledTrigger-10841-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3091934 INFO  (qtp1878175356-33305) [n:127.0.0.1:41041_i_%2Fmh c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 3091965 INFO  (qtp1878175356-33305) [n:127.0.0.1:41041_i_%2Fmh c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.IndexSchema [collection1_shard2_replica_t41] Schema name=test
   [junit4]   2> 3092049 INFO  (qtp1878175356-33305) [n:127.0.0.1:41041_i_%2Fmh c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 3092066 INFO  (qtp1878175356-33305) [n:127.0.0.1:41041_i_%2Fmh c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_t41' using configuration from collection collection1, trusted=true
   [junit4]   2> 3092067 INFO  (qtp1878175356-33305) [n:127.0.0.1:41041_i_%2Fmh c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_t41' (registry 'solr.core.collection1.shard2.replica_t41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68aba492
   [junit4]   2> 3092067 INFO  (qtp1878175356-33305) [n:127.0.0.1:41041_i_%2Fmh c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SolrCore [[collection1_shard2_replica_t41] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_6CBEDD2BE0D3C029-001/shard-1-001/cores/collection1_shard2_replica_t41], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_6CBEDD2BE0D3C029-001/shard-1-001/cores/collection1_shard2_replica_t41/data/]
   [junit4]   2> 3092072 INFO  (qtp1878175356-33305) [n:127.0.0.1:41041_i_%2Fmh c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=81.166015625, floorSegmentMB=1.3681640625, forceMergeDeletesPctAllowed=14.083262919000399, segmentsPerTier=44.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0, deletesPctAllowed=41.514864962384394
   [junit4]   2> 3092081 WARN  (qtp1878175356-33305) [n:127.0.0.1:41041_i_%2Fmh c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3092143 INFO  (qtp1878175356-33305) [n:127.0.0.1:41041_i_%2Fmh c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3092143 INFO  (qtp1878175356-33305) [n:127.0.0.1:41041_i_%2Fmh c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3092144 INFO  (qtp1878175356-33305) [n:127.0.0.1:41041_i_%2Fmh c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
   [junit4]   2> 3092144 INFO  (qtp1878175356-33305) [n:127.0.0.1:41041_i_%2Fmh c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3092146 INFO  (qtp1878175356-33305) [n:127.0.0.1:41041_i_%2Fmh c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@523363d8
   [junit4]   2> 3092146 INFO  (qtp1878175356-33305) [n:127.0.0.1:41041_i_%2Fmh c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.SolrIndexSearcher Opening [Searcher@5ee1959c[collection1_shard2_replica_t41] main]
   [junit4]   2> 3092148 INFO  (qtp1878175356-33305) [n:127.0.0.1:41041_i_%2Fmh c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3092148 INFO  (qtp1878175356-33305) [n:127.0.0.1:41041_i_%2Fmh c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]  

[...truncated too long message...]

vy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null88439078
     [copy] Copying 238 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null88439078
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null88439078

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 36ms :: artifacts dl 2ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 307 minutes 53 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-NightlyTests-master - Build # 33 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-master/33/

6 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest:     1) Thread[id=223856, name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[9E21EF6315D422A4]-EventThread, state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)    2) Thread[id=223854, name=Connection evictor, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=223857, name=zkConnectionManagerCallback-10531-thread-1, state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=223855, name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[9E21EF6315D422A4]-SendThread(127.0.0.1:33431), state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest:
   1) Thread[id=223856, name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[9E21EF6315D422A4]-EventThread, state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
   2) Thread[id=223854, name=Connection evictor, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=223857, name=zkConnectionManagerCallback-10531-thread-1, state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=223855, name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[9E21EF6315D422A4]-SendThread(127.0.0.1:33431), state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
        at __randomizedtesting.SeedInfo.seed([9E21EF6315D422A4]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=223857, name=zkConnectionManagerCallback-10531-thread-1, state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=223855, name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[9E21EF6315D422A4]-SendThread(127.0.0.1:33431), state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=223857, name=zkConnectionManagerCallback-10531-thread-1, state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=223855, name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[9E21EF6315D422A4]-SendThread(127.0.0.1:33431), state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
        at __randomizedtesting.SeedInfo.seed([9E21EF6315D422A4]:0)


FAILED:  org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test

Error Message:
Error from server at <a href="http://127.0.0.1:46188:">http://127.0.0.1:46188: ADDREPLICA failed to create replica

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at <a href="http://127.0.0.1:46188:">http://127.0.0.1:46188: ADDREPLICA failed to create replica
        at __randomizedtesting.SeedInfo.seed([9E21EF6315D422A4:1675D0B9BB284F5C]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.UnloadDistributedZkTest.testUnloadShardAndCollection(UnloadDistributedZkTest.java:127)
        at org.apache.solr.cloud.UnloadDistributedZkTest.test(UnloadDistributedZkTest.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting

Error Message:
Error from server at https://127.0.0.1:35157/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason: <pre>    Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>  </body> </html>

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://127.0.0.1:35157/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at __randomizedtesting.SeedInfo.seed([D35431B6B924A499:11E30DDEBA6454E1]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:237)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting(CloudSolrClientTest.java:269)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:35157/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more


FAILED:  org.apache.solr.cloud.LIROnShardRestartTest.testSeveralReplicasInLIR

Error Message:
Timeout waiting for replicas rejoin election

Stack Trace:
java.lang.AssertionError: Timeout waiting for replicas rejoin election
        at __randomizedtesting.SeedInfo.seed([9E21EF6315D422A4:187B7857358E6FFD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertFalse(Assert.java:68)
        at org.apache.solr.cloud.LIROnShardRestartTest.testSeveralReplicasInLIR(LIROnShardRestartTest.java:242)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:40122/gcxlx/l/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:40122/gcxlx/l/collection1
        at __randomizedtesting.SeedInfo.seed([9E21EF6315D422A4:1675D0B9BB284F5C]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:504)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:479)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:1590)
        at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:215)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 53 more




Build Log:
[...truncated 14227 lines...]
   [junit4] Suite: org.apache.solr.cloud.LIROnShardRestartTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/init-core-data-001
   [junit4]   2> 3200889 WARN  (SUITE-LIROnShardRestartTest-seed#[9E21EF6315D422A4]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=59 numCloses=59
   [junit4]   2> 3200889 INFO  (SUITE-LIROnShardRestartTest-seed#[9E21EF6315D422A4]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 3200892 INFO  (SUITE-LIROnShardRestartTest-seed#[9E21EF6315D422A4]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 3200893 INFO  (SUITE-LIROnShardRestartTest-seed#[9E21EF6315D422A4]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 3200893 INFO  (SUITE-LIROnShardRestartTest-seed#[9E21EF6315D422A4]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001
   [junit4]   2> 3200894 INFO  (SUITE-LIROnShardRestartTest-seed#[9E21EF6315D422A4]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3200894 INFO  (Thread-11139) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3200894 INFO  (Thread-11139) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3200899 ERROR (Thread-11139) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3200994 INFO  (SUITE-LIROnShardRestartTest-seed#[9E21EF6315D422A4]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:44569
   [junit4]   2> 3201007 INFO  (zkConnectionManagerCallback-6531-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3201028 INFO  (jetty-launcher-6528-thread-3) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 3201028 INFO  (jetty-launcher-6528-thread-1) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 3201029 INFO  (jetty-launcher-6528-thread-2) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 3201029 INFO  (jetty-launcher-6528-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3201029 INFO  (jetty-launcher-6528-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3201029 INFO  (jetty-launcher-6528-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 3201030 INFO  (jetty-launcher-6528-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4409cd20{/solr,null,AVAILABLE}
   [junit4]   2> 3201030 INFO  (jetty-launcher-6528-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3201030 INFO  (jetty-launcher-6528-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3201030 INFO  (jetty-launcher-6528-thread-3) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 3201031 INFO  (jetty-launcher-6528-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7f76f010{SSL,[ssl, http/1.1]}{127.0.0.1:35418}
   [junit4]   2> 3201031 INFO  (jetty-launcher-6528-thread-1) [    ] o.e.j.s.Server Started @3201093ms
   [junit4]   2> 3201031 INFO  (jetty-launcher-6528-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35418}
   [junit4]   2> 3201031 ERROR (jetty-launcher-6528-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3201031 INFO  (jetty-launcher-6528-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3201031 INFO  (jetty-launcher-6528-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 3201031 INFO  (jetty-launcher-6528-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3201031 INFO  (jetty-launcher-6528-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3201031 INFO  (jetty-launcher-6528-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-19T21:01:05.097Z
   [junit4]   2> 3201032 INFO  (jetty-launcher-6528-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3201032 INFO  (jetty-launcher-6528-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3201032 INFO  (jetty-launcher-6528-thread-2) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 3201034 INFO  (jetty-launcher-6528-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3486d04{/solr,null,AVAILABLE}
   [junit4]   2> 3201034 INFO  (jetty-launcher-6528-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@e949ec0{/solr,null,AVAILABLE}
   [junit4]   2> 3201035 INFO  (jetty-launcher-6528-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4ce7d057{SSL,[ssl, http/1.1]}{127.0.0.1:35452}
   [junit4]   2> 3201035 INFO  (jetty-launcher-6528-thread-2) [    ] o.e.j.s.Server Started @3201097ms
   [junit4]   2> 3201035 INFO  (jetty-launcher-6528-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@77f1bb11{SSL,[ssl, http/1.1]}{127.0.0.1:44709}
   [junit4]   2> 3201035 INFO  (jetty-launcher-6528-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35452}
   [junit4]   2> 3201035 INFO  (jetty-launcher-6528-thread-3) [    ] o.e.j.s.Server Started @3201097ms
   [junit4]   2> 3201035 INFO  (jetty-launcher-6528-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44709}
   [junit4]   2> 3201035 ERROR (jetty-launcher-6528-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3201035 INFO  (jetty-launcher-6528-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3201035 ERROR (jetty-launcher-6528-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3201035 INFO  (jetty-launcher-6528-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 3201035 INFO  (jetty-launcher-6528-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3201035 INFO  (jetty-launcher-6528-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3201035 INFO  (jetty-launcher-6528-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 3201035 INFO  (jetty-launcher-6528-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3201035 INFO  (jetty-launcher-6528-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3201035 INFO  (jetty-launcher-6528-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-19T21:01:05.101Z
   [junit4]   2> 3201035 INFO  (jetty-launcher-6528-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3201035 INFO  (jetty-launcher-6528-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-19T21:01:05.101Z
   [junit4]   2> 3201036 INFO  (zkConnectionManagerCallback-6533-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3201037 INFO  (zkConnectionManagerCallback-6535-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3201037 INFO  (jetty-launcher-6528-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3201038 INFO  (zkConnectionManagerCallback-6537-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3201038 INFO  (jetty-launcher-6528-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3201041 INFO  (jetty-launcher-6528-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3201096 INFO  (jetty-launcher-6528-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44569/solr
   [junit4]   2> 3201098 INFO  (zkConnectionManagerCallback-6541-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3201100 INFO  (zkConnectionManagerCallback-6543-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3201155 INFO  (jetty-launcher-6528-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44569/solr
   [junit4]   2> 3201157 INFO  (zkConnectionManagerCallback-6549-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3201160 INFO  (zkConnectionManagerCallback-6551-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3201174 INFO  (jetty-launcher-6528-thread-3) [n:127.0.0.1:44709_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:44709_solr
   [junit4]   2> 3201175 INFO  (jetty-launcher-6528-thread-3) [n:127.0.0.1:44709_solr    ] o.a.s.c.Overseer Overseer (id=72130840437063687-127.0.0.1:44709_solr-n_0000000000) starting
   [junit4]   2> 3201187 INFO  (zkConnectionManagerCallback-6558-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3201190 INFO  (jetty-launcher-6528-thread-3) [n:127.0.0.1:44709_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44569/solr ready
   [junit4]   2> 3201191 INFO  (OverseerStateUpdate-72130840437063687-127.0.0.1:44709_solr-n_0000000000) [n:127.0.0.1:44709_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:44709_solr
   [junit4]   2> 3201192 INFO  (jetty-launcher-6528-thread-3) [n:127.0.0.1:44709_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44709_solr
   [junit4]   2> 3201193 DEBUG (OverseerAutoScalingTriggerThread-72130840437063687-127.0.0.1:44709_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 3201197 INFO  (OverseerStateUpdate-72130840437063687-127.0.0.1:44709_solr-n_0000000000) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3201207 DEBUG (OverseerAutoScalingTriggerThread-72130840437063687-127.0.0.1:44709_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 3201207 DEBUG (OverseerAutoScalingTriggerThread-72130840437063687-127.0.0.1:44709_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 3201207 DEBUG (OverseerAutoScalingTriggerThread-72130840437063687-127.0.0.1:44709_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 3201215 INFO  (zkCallback-6557-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3201225 DEBUG (OverseerAutoScalingTriggerThread-72130840437063687-127.0.0.1:44709_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:44709_solr]
   [junit4]   2> 3201225 DEBUG (OverseerAutoScalingTriggerThread-72130840437063687-127.0.0.1:44709_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3201226 DEBUG (OverseerAutoScalingTriggerThread-72130840437063687-127.0.0.1:44709_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 3201227 DEBUG (ScheduledTrigger-9344-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 3201229 INFO  (jetty-launcher-6528-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44569/solr
   [junit4]   2> 3201232 INFO  (zkConnectionManagerCallback-6562-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3201236 INFO  (jetty-launcher-6528-thread-3) [n:127.0.0.1:44709_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3201236 INFO  (zkConnectionManagerCallback-6565-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3201248 INFO  (jetty-launcher-6528-thread-1) [n:127.0.0.1:35418_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3201258 INFO  (jetty-launcher-6528-thread-2) [n:127.0.0.1:35452_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3201260 INFO  (jetty-launcher-6528-thread-1) [n:127.0.0.1:35418_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3201260 INFO  (jetty-launcher-6528-thread-1) [n:127.0.0.1:35418_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35418_solr
   [junit4]   2> 3201261 INFO  (zkCallback-6557-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3201265 INFO  (zkCallback-6550-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3201275 INFO  (jetty-launcher-6528-thread-2) [n:127.0.0.1:35452_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3201276 INFO  (jetty-launcher-6528-thread-2) [n:127.0.0.1:35452_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35452_solr
   [junit4]   2> 3201277 INFO  (zkCallback-6557-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3201277 INFO  (zkCallback-6550-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3201282 INFO  (jetty-launcher-6528-thread-3) [n:127.0.0.1:44709_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44709.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36da70f8
   [junit4]   2> 3201283 INFO  (zkCallback-6563-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 3201289 INFO  (zkCallback-6542-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 3201294 INFO  (zkConnectionManagerCallback-6572-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3201296 INFO  (jetty-launcher-6528-thread-3) [n:127.0.0.1:44709_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44709.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36da70f8
   [junit4]   2> 3201296 INFO  (jetty-launcher-6528-thread-3) [n:127.0.0.1:44709_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44709.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36da70f8
   [junit4]   2> 3201298 INFO  (jetty-launcher-6528-thread-3) [n:127.0.0.1:44709_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/.
   [junit4]   2> 3201299 INFO  (jetty-launcher-6528-thread-1) [n:127.0.0.1:35418_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 3201300 INFO  (jetty-launcher-6528-thread-1) [n:127.0.0.1:35418_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44569/solr ready
   [junit4]   2> 3201302 INFO  (jetty-launcher-6528-thread-1) [n:127.0.0.1:35418_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3201303 INFO  (zkConnectionManagerCallback-6578-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3201304 INFO  (jetty-launcher-6528-thread-2) [n:127.0.0.1:35452_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 3201305 INFO  (jetty-launcher-6528-thread-2) [n:127.0.0.1:35452_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44569/solr ready
   [junit4]   2> 3201306 INFO  (jetty-launcher-6528-thread-2) [n:127.0.0.1:35452_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3201360 INFO  (jetty-launcher-6528-thread-2) [n:127.0.0.1:35452_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35452.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36da70f8
   [junit4]   2> 3201378 INFO  (jetty-launcher-6528-thread-1) [n:127.0.0.1:35418_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35418.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36da70f8
   [junit4]   2> 3201381 INFO  (jetty-launcher-6528-thread-2) [n:127.0.0.1:35452_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35452.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36da70f8
   [junit4]   2> 3201381 INFO  (jetty-launcher-6528-thread-2) [n:127.0.0.1:35452_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35452.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36da70f8
   [junit4]   2> 3201383 INFO  (jetty-launcher-6528-thread-2) [n:127.0.0.1:35452_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node2/.
   [junit4]   2> 3201393 INFO  (jetty-launcher-6528-thread-1) [n:127.0.0.1:35418_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35418.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36da70f8
   [junit4]   2> 3201396 INFO  (jetty-launcher-6528-thread-1) [n:127.0.0.1:35418_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35418.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36da70f8
   [junit4]   2> 3201397 INFO  (jetty-launcher-6528-thread-1) [n:127.0.0.1:35418_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node1/.
   [junit4]   2> 3201481 INFO  (zkConnectionManagerCallback-6581-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3201486 INFO  (zkConnectionManagerCallback-6586-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3201487 INFO  (SUITE-LIROnShardRestartTest-seed#[9E21EF6315D422A4]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 3201488 INFO  (SUITE-LIROnShardRestartTest-seed#[9E21EF6315D422A4]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44569/solr ready
   [junit4]   2> 3201808 INFO  (TEST-LIROnShardRestartTest.testAllReplicasInLIR-seed#[9E21EF6315D422A4]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testAllReplicasInLIR
   [junit4]   2> 3201843 INFO  (qtp506345389-33409) [n:127.0.0.1:35452_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&name=allReplicasInLIR&nrtReplicas=3&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3201846 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection allReplicasInLIR
   [junit4]   2> 3201847 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/managed-schema to /configs/allReplicasInLIR.AUTOCREATED/managed-schema
   [junit4]   2> 3201850 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/protwords.txt to /configs/allReplicasInLIR.AUTOCREATED/protwords.txt
   [junit4]   2> 3201852 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/solrconfig.xml to /configs/allReplicasInLIR.AUTOCREATED/solrconfig.xml
   [junit4]   2> 3201854 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/synonyms.txt to /configs/allReplicasInLIR.AUTOCREATED/synonyms.txt
   [junit4]   2> 3201856 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/stopwords.txt to /configs/allReplicasInLIR.AUTOCREATED/stopwords.txt
   [junit4]   2> 3201858 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_it.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/contractions_it.txt
   [junit4]   2> 3201861 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ca.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/contractions_ca.txt
   [junit4]   2> 3201863 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stemdict_nl.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stemdict_nl.txt
   [junit4]   2> 3201865 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_no.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_no.txt
   [junit4]   2> 3201868 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hy.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_hy.txt
   [junit4]   2> 3201870 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_id.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_id.txt
   [junit4]   2> 3201875 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hi.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_hi.txt
   [junit4]   2> 3201877 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ga.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_ga.txt
   [junit4]   2> 3201880 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_da.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_da.txt
   [junit4]   2> 3201882 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ja.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_ja.txt
   [junit4]   2> 3201884 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_pt.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_pt.txt
   [junit4]   2> 3201886 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hu.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_hu.txt
   [junit4]   2> 3201888 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_el.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_el.txt
   [junit4]   2> 3201890 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ru.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_ru.txt
   [junit4]   2> 3201893 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_tr.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_tr.txt
   [junit4]   2> 3201895 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ar.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_ar.txt
   [junit4]   2> 3201897 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/userdict_ja.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/userdict_ja.txt
   [junit4]   2> 3201900 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_eu.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_eu.txt
   [junit4]   2> 3201902 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_it.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_it.txt
   [junit4]   2> 3201904 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_cz.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_cz.txt
   [junit4]   2> 3201906 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stoptags_ja.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stoptags_ja.txt
   [junit4]   2> 3201908 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_fr.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/contractions_fr.txt
   [junit4]   2> 3201911 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_de.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_de.txt
   [junit4]   2> 3201913 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fa.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_fa.txt
   [junit4]   2> 3201916 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ga.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/contractions_ga.txt
   [junit4]   2> 3201918 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ca.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_ca.txt
   [junit4]   2> 3201920 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_nl.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_nl.txt
   [junit4]   2> 3201923 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_bg.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_bg.txt
   [junit4]   2> 3201925 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_en.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_en.txt
   [junit4]   2> 3201927 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_es.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_es.txt
   [junit4]   2> 3201930 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_lv.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_lv.txt
   [junit4]   2> 3201932 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ro.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_ro.txt
   [junit4]   2> 3201934 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_th.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_th.txt
   [junit4]   2> 3201936 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fr.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_fr.txt
   [junit4]   2> 3201939 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_sv.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_sv.txt
   [junit4]   2> 3201941 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fi.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_fi.txt
   [junit4]   2> 3201943 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/hyphenations_ga.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/hyphenations_ga.txt
   [junit4]   2> 3201946 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_gl.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_gl.txt
   [junit4]   2> 3201948 INFO  (OverseerThreadFactory-9346-thread-1-processing-n:127.0.0.1:44709_solr) [n:127.0.0.1:44709_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/params.json to /configs/allReplicasInLIR.AUTOCREATED/params.json
   [junit4]   2> 3202070 INFO  (qtp506345389-33410) [n:127.0.0.1:35452_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3202110 INFO  (qtp1051231833-33396) [n:127.0.0.1:35418_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3202140 INFO  (qtp805193647-33392) [n:127.0.0.1:44709_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3202146 INFO  (OverseerStateUpdate-72130840437063687-127.0.0.1:44709_solr-n_0000000000) [n:127.0.0.1:44709_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"allReplicasInLIR",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"allReplicasInLIR_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:44709/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 3202149 INFO  (OverseerStateUpdate-72130840437063687-127.0.0.1:44709_solr-n_0000000000) [n:127.0.0.1:44709_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"allReplicasInLIR",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"allReplicasInLIR_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:35418/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 3202152 INFO  (OverseerStateUpdate-72130840437063687-127.0.0.1:44709_solr-n_0000000000) [n:127.0.0.1:44709_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"allReplicasInLIR",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"allReplicasInLIR_shard1_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:35452/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 3202233 DEBUG (ScheduledTrigger-9344-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 3202364 INFO  (qtp805193647-33392) [n:127.0.0.1:44709_solr    x:allReplicasInLIR_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=allReplicasInLIR.AUTOCREATED&newCollection=true&name=allReplicasInLIR_shard1_replica_n1&action=CREATE&numShards=1&collection=allReplicasInLIR&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3202364 INFO  (qtp805193647-33392) [n:127.0.0.1:44709_solr    x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3202365 INFO  (qtp506345389-33410) [n:127.0.0.1:35452_solr    x:allReplicasInLIR_shard1_replica_n4] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=allReplicasInLIR.AUTOCREATED&newCollection=true&name=allReplicasInLIR_shard1_replica_n4&action=CREATE&numShards=1&collection=allReplicasInLIR&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3202366 INFO  (qtp1051231833-33396) [n:127.0.0.1:35418_solr    x:allReplicasInLIR_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=allReplicasInLIR.AUTOCREATED&newCollection=true&name=allReplicasInLIR_shard1_replica_n2&action=CREATE&numShards=1&collection=allReplicasInLIR&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3203234 DEBUG (ScheduledTrigger-9344-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 3203391 INFO  (qtp506345389-33410) [n:127.0.0.1:35452_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n4] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
   [junit4]   2> 3203391 INFO  (qtp506345389-33410) [n:127.0.0.1:35452_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n4] o.a.s.c.RequestParams request params refreshed to version 0
   [junit4]   2> 3203391 INFO  (qtp805193647-33392) [n:127.0.0.1:44709_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
   [junit4]   2> 3203391 INFO  (qtp1051231833-33396) [n:127.0.0.1:35418_solr c:allReplicasInLIR s:shard1 r:core_node5 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
   [junit4]   2> 3203391 INFO  (qtp805193647-33392) [n:127.0.0.1:44709_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.RequestParams request params refreshed to version 0
   [junit4]   2> 3203391 INFO  (qtp1051231833-33396) [n:127.0.0.1:35418_solr c:allReplicasInLIR s:shard1 r:core_node5 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.RequestParams request params refreshed to version 0
   [junit4]   2> 3203392 WARN  (qtp506345389-33410) [n:127.0.0.1:35452_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n4] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n4/../../../../contrib/extraction/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n4/../../../../contrib/extraction/lib
   [junit4]   2> 3203392 WARN  (qtp805193647-33392) [n:127.0.0.1:44709_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../contrib/extraction/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../contrib/extraction/lib
   [junit4]   2> 3203392 WARN  (qtp1051231833-33396) [n:127.0.0.1:35418_solr c:allReplicasInLIR s:shard1 r:core_node5 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n2/../../../../contrib/extraction/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n2/../../../../contrib/extraction/lib
   [junit4]   2> 3203396 WARN  (qtp805193647-33392) [n:127.0.0.1:44709_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../dist filtered by solr-cell-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../dist
   [junit4]   2> 3203396 WARN  (qtp1051231833-33396) [n:127.0.0.1:35418_solr c:allReplicasInLIR s:shard1 r:core_node5 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n2/../../../../dist filtered by solr-cell-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n2/../../../../dist
   [junit4]   2> 3203397 WARN  (qtp805193647-33392) [n:127.0.0.1:44709_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../contrib/clustering/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../contrib/clustering/lib
   [junit4]   2> 3203397 WARN  (qtp1051231833-33396) [n:127.0.0.1:35418_solr c:allReplicasInLIR s:shard1 r:core_node5 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n2/../../../../contrib/clustering/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n2/../../../../contrib/clustering/lib
   [junit4]   2> 3203397 WARN  (qtp506345389-33410) [n:127.0.0.1:35452_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n4] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n4/../../../../dist filtered by solr-cell-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n4/../../../../dist
   [junit4]   2> 3203397 WARN  (qtp805193647-33392) [n:127.0.0.1:44709_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../dist filtered by solr-clustering-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../dist
   [junit4]   2> 3203397 WARN  (qtp1051231833-33396) [n:127.0.0.1:35418_solr c:allReplicasInLIR s:shard1 r:core_node5 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n2/../../../../dist filtered by solr-clustering-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n2/../../../../dist
   [junit4]   2> 3203397 WARN  (qtp506345389-33410) [n:127.0.0.1:35452_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n4] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n4/../../../../contrib/clustering/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n4/../../../../contrib/clustering/lib
   [junit4]   2> 3203397 WARN  (qtp805193647-33392) [n:127.0.0.1:44709_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../contrib/langid/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../contrib/langid/lib
   [junit4]   2> 3203397 WARN  (qtp1051231833-33396) [n:127.0.0.1:35418_solr c:allReplicasInLIR s:shard1 r:core_node5 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n2/../../../../contrib/langid/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n2/../../../../contrib/langid/lib
   [junit4]   2> 3203397 WARN  (qtp506345389-33410) [n:127.0.0.1:35452_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n4] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n4/../../../../dist filtered by solr-clustering-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n4/../../../../dist
   [junit4]   2> 3203398 WARN  (qtp805193647-33392) [n:127.0.0.1:44709_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../dist filtered by solr-langid-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../dist
   [junit4]   2> 3203398 WARN  (qtp1051231833-33396) [n:127.0.0.1:35418_solr c:allReplicasInLIR s:shard1 r:core_node5 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n2/../../../../dist filtered by solr-langid-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n2/../../../../dist
   [junit4]   2> 3203398 WARN  (qtp506345389-33410) [n:127.0.0.1:35452_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n4] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n4/../../../../contrib/langid/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n4/../../../../contrib/langid/lib
   [junit4]   2> 3203398 WARN  (qtp1051231833-33396) [n:127.0.0.1:35418_solr c:allReplicasInLIR s:shard1 r:core_node5 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n2/../../../../contrib/velocity/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n2/../../../../contrib/velocity/lib
   [junit4]   2> 3203398 WARN  (qtp805193647-33392) [n:127.0.0.1:44709_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../contrib/velocity/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../contrib/velocity/lib
   [junit4]   2> 3203398 WARN  (qtp506345389-33410) [n:127.0.0.1:35452_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n4] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n4/../../../../dist filtered by solr-langid-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n4/../../../../dist
   [junit4]   2> 3203398 WARN  (qtp805193647-33392) [n:127.0.0.1:44709_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../dist filtered by solr-velocity-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../dist
   [junit4]   2> 3203398 WARN  (qtp506345389-33410) [n:127.0.0.1:35452_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n4] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n4/../../../../contrib/velocity/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n4/../../../../contrib/velocity/lib
   [junit4]   2> 3203398 WARN  (qtp805193647-33392) [n:127.0.0.1:44709_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../dist filtered by solr-ltr-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.LIROnShardRestartTest_9E21EF6315D422A4-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n1/../../../../dist
   [junit4]   2> 3203399 WARN  (qtp506345389-33410) [n:127.0.0.1:35452_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n4] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/w

[...truncated too long message...]

configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null643364820
     [copy] Copying 240 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null643364820
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/null643364820

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 36ms :: artifacts dl 4ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 326 minutes 36 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]