[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1330 - Unstable

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1330 - Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1330/

4 tests failed.
FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
Doc with id=934 not found in http://127.0.0.1:50086/hm_ymx/c8n_1x2 due to: Path not found: /id; rsp={doc=null}

Stack Trace:
java.lang.AssertionError: Doc with id=934 not found in http://127.0.0.1:50086/hm_ymx/c8n_1x2 due to: Path not found: /id; rsp={doc=null}
        at __randomizedtesting.SeedInfo.seed([6680E89DBC9BEEC3:EED4D7471267833B]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.HttpPartitionTest.assertDocExists(HttpPartitionTest.java:602)
        at org.apache.solr.cloud.HttpPartitionTest.assertDocsExistInAllReplicas(HttpPartitionTest.java:557)
        at org.apache.solr.cloud.HttpPartitionTest.testRf2(HttpPartitionTest.java:378)
        at org.apache.solr.cloud.HttpPartitionTest.test(HttpPartitionTest.java:131)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.rule.RulesTest.doIntegrationTest

Error Message:
Error from server at http://127.0.0.1:38802/solr: delete the collection time out:180s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:38802/solr: delete the collection time out:180s
        at __randomizedtesting.SeedInfo.seed([6680E89DBC9BEEC3:83B3AF1CA0EF1CC1]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:595)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:231)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:220)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:459)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:389)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1135)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:876)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:807)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.MiniSolrCloudCluster.deleteAllCollections(MiniSolrCloudCluster.java:442)
        at org.apache.solr.cloud.rule.RulesTest.removeCollections(RulesTest.java:63)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:965)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.rule.RulesTest.testModifyColl

Error Message:
Error from server at http://127.0.0.1:38802/solr: Could not find collection : rulesColl

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:38802/solr: Could not find collection : rulesColl
        at __randomizedtesting.SeedInfo.seed([6680E89DBC9BEEC3:CF18C2E8B0572AC5]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:595)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:231)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:220)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:459)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:389)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1135)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:876)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:807)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.MiniSolrCloudCluster.deleteAllCollections(MiniSolrCloudCluster.java:442)
        at org.apache.solr.cloud.rule.RulesTest.removeCollections(RulesTest.java:63)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:965)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testArraySort

Error Message:
--> https://127.0.0.1:34427/solr/collection1:Invalid stream expression arraySort(array(11.5,12.3,4,3,1,0)) - function 'arraySort' is unknown (not mapped to a valid TupleStream)

Stack Trace:
java.io.IOException: --> https://127.0.0.1:34427/solr/collection1:Invalid stream expression arraySort(array(11.5,12.3,4,3,1,0)) - function 'arraySort' is unknown (not mapped to a valid TupleStream)
        at __randomizedtesting.SeedInfo.seed([29C234860309F872:1F74E2AB84D0040B]:0)
        at org.apache.solr.client.solrj.io.stream.SolrStream.read(SolrStream.java:219)
        at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.getTuples(StreamExpressionTest.java:7487)
        at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testArraySort(StreamExpressionTest.java:5840)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12272 lines...]
   [junit4] Suite: org.apache.solr.cloud.rule.RulesTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.rule.RulesTest_6680E89DBC9BEEC3-001/init-core-data-001
   [junit4]   2> 920913 WARN  (SUITE-RulesTest-seed#[6680E89DBC9BEEC3]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=16 numCloses=16
   [junit4]   2> 920913 INFO  (SUITE-RulesTest-seed#[6680E89DBC9BEEC3]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 920915 INFO  (SUITE-RulesTest-seed#[6680E89DBC9BEEC3]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 920915 INFO  (SUITE-RulesTest-seed#[6680E89DBC9BEEC3]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 5 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.rule.RulesTest_6680E89DBC9BEEC3-001/tempDir-001
   [junit4]   2> 920915 INFO  (SUITE-RulesTest-seed#[6680E89DBC9BEEC3]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 920915 INFO  (Thread-3104) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 920915 INFO  (Thread-3104) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 920917 ERROR (Thread-3104) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 921015 INFO  (SUITE-RulesTest-seed#[6680E89DBC9BEEC3]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:46058
   [junit4]   2> 921026 INFO  (jetty-launcher-2106-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 921026 INFO  (jetty-launcher-2106-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 921026 INFO  (jetty-launcher-2106-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 921027 INFO  (jetty-launcher-2106-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 921027 INFO  (jetty-launcher-2106-thread-5) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 921053 INFO  (jetty-launcher-2106-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@699db8fc{/solr,null,AVAILABLE}
   [junit4]   2> 921053 INFO  (jetty-launcher-2106-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1730e760{HTTP/1.1,[http/1.1]}{127.0.0.1:35640}
   [junit4]   2> 921053 INFO  (jetty-launcher-2106-thread-1) [    ] o.e.j.s.Server Started @923504ms
   [junit4]   2> 921053 INFO  (jetty-launcher-2106-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35640}
   [junit4]   2> 921053 ERROR (jetty-launcher-2106-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 921054 INFO  (jetty-launcher-2106-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 921054 INFO  (jetty-launcher-2106-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 921054 INFO  (jetty-launcher-2106-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 921054 INFO  (jetty-launcher-2106-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T04:53:41.324Z
   [junit4]   2> 921056 INFO  (jetty-launcher-2106-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@509c6811{/solr,null,AVAILABLE}
   [junit4]   2> 921056 INFO  (jetty-launcher-2106-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3ea93b20{HTTP/1.1,[http/1.1]}{127.0.0.1:38802}
   [junit4]   2> 921056 INFO  (jetty-launcher-2106-thread-3) [    ] o.e.j.s.Server Started @923507ms
   [junit4]   2> 921056 INFO  (jetty-launcher-2106-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38802}
   [junit4]   2> 921056 ERROR (jetty-launcher-2106-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 921056 INFO  (jetty-launcher-2106-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 921056 INFO  (jetty-launcher-2106-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 921057 INFO  (jetty-launcher-2106-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 921057 INFO  (jetty-launcher-2106-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T04:53:41.327Z
   [junit4]   2> 921060 INFO  (jetty-launcher-2106-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@42dde8fc{/solr,null,AVAILABLE}
   [junit4]   2> 921061 INFO  (jetty-launcher-2106-thread-5) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@12cd7a2d{/solr,null,AVAILABLE}
   [junit4]   2> 921062 INFO  (jetty-launcher-2106-thread-5) [    ] o.e.j.s.AbstractConnector Started ServerConnector@62065457{HTTP/1.1,[http/1.1]}{127.0.0.1:55312}
   [junit4]   2> 921062 INFO  (jetty-launcher-2106-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5d1b2111{HTTP/1.1,[http/1.1]}{127.0.0.1:53043}
   [junit4]   2> 921062 INFO  (jetty-launcher-2106-thread-4) [    ] o.e.j.s.Server Started @923513ms
   [junit4]   2> 921062 INFO  (jetty-launcher-2106-thread-5) [    ] o.e.j.s.Server Started @923513ms
   [junit4]   2> 921062 INFO  (jetty-launcher-2106-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53043}
   [junit4]   2> 921062 INFO  (jetty-launcher-2106-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55312}
   [junit4]   2> 921062 ERROR (jetty-launcher-2106-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 921062 ERROR (jetty-launcher-2106-thread-5) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 921062 INFO  (jetty-launcher-2106-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 921062 INFO  (jetty-launcher-2106-thread-5) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 921062 INFO  (jetty-launcher-2106-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 921062 INFO  (jetty-launcher-2106-thread-5) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 921062 INFO  (jetty-launcher-2106-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 921062 INFO  (jetty-launcher-2106-thread-5) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 921062 INFO  (jetty-launcher-2106-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T04:53:41.332Z
   [junit4]   2> 921062 INFO  (jetty-launcher-2106-thread-5) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T04:53:41.332Z
   [junit4]   2> 921063 INFO  (jetty-launcher-2106-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6d17b7c{/solr,null,AVAILABLE}
   [junit4]   2> 921064 INFO  (jetty-launcher-2106-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1d3e742{HTTP/1.1,[http/1.1]}{127.0.0.1:60191}
   [junit4]   2> 921064 INFO  (jetty-launcher-2106-thread-2) [    ] o.e.j.s.Server Started @923515ms
   [junit4]   2> 921064 INFO  (jetty-launcher-2106-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=60191}
   [junit4]   2> 921064 ERROR (jetty-launcher-2106-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 921064 INFO  (jetty-launcher-2106-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 921064 INFO  (jetty-launcher-2106-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 921064 INFO  (jetty-launcher-2106-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 921064 INFO  (jetty-launcher-2106-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T04:53:41.334Z
   [junit4]   2> 921066 INFO  (jetty-launcher-2106-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 921070 INFO  (jetty-launcher-2106-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 921074 INFO  (jetty-launcher-2106-thread-5) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 921075 INFO  (jetty-launcher-2106-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 921078 INFO  (jetty-launcher-2106-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 921079 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cbeb300c20005, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 921079 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cbeb300c20003, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 921080 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cbeb300c20001, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 921083 INFO  (jetty-launcher-2106-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46058/solr
   [junit4]   2> 921084 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cbeb300c20002, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 921097 INFO  (jetty-launcher-2106-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46058/solr
   [junit4]   2> 921097 INFO  (jetty-launcher-2106-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46058/solr
   [junit4]   2> 921099 INFO  (jetty-launcher-2106-thread-5) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46058/solr
   [junit4]   2> 921100 INFO  (jetty-launcher-2106-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46058/solr
   [junit4]   2> 921106 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cbeb300c2000a, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 921147 INFO  (jetty-launcher-2106-thread-1) [n:127.0.0.1:35640_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 921147 INFO  (jetty-launcher-2106-thread-5) [n:127.0.0.1:55312_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 921147 INFO  (jetty-launcher-2106-thread-1) [n:127.0.0.1:35640_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35640_solr
   [junit4]   2> 921148 INFO  (jetty-launcher-2106-thread-3) [n:127.0.0.1:38802_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 921148 INFO  (jetty-launcher-2106-thread-5) [n:127.0.0.1:55312_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55312_solr
   [junit4]   2> 921148 INFO  (jetty-launcher-2106-thread-2) [n:127.0.0.1:60191_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 921148 INFO  (jetty-launcher-2106-thread-4) [n:127.0.0.1:53043_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 921148 INFO  (jetty-launcher-2106-thread-1) [n:127.0.0.1:35640_solr    ] o.a.s.c.Overseer Overseer (id=98162967916445708-127.0.0.1:35640_solr-n_0000000000) starting
   [junit4]   2> 921149 INFO  (zkCallback-2132-thread-1-processing-n:127.0.0.1:55312_solr) [n:127.0.0.1:55312_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 921150 INFO  (zkCallback-2130-thread-1-processing-n:127.0.0.1:35640_solr) [n:127.0.0.1:35640_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 921151 INFO  (zkCallback-2128-thread-1-processing-n:127.0.0.1:38802_solr) [n:127.0.0.1:38802_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 921152 INFO  (zkCallback-2133-thread-1-processing-n:127.0.0.1:53043_solr) [n:127.0.0.1:53043_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 921153 INFO  (zkCallback-2131-thread-1-processing-n:127.0.0.1:60191_solr) [n:127.0.0.1:60191_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 921157 INFO  (jetty-launcher-2106-thread-3) [n:127.0.0.1:38802_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38802_solr
   [junit4]   2> 921157 INFO  (jetty-launcher-2106-thread-2) [n:127.0.0.1:60191_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60191_solr
   [junit4]   2> 921160 INFO  (jetty-launcher-2106-thread-4) [n:127.0.0.1:53043_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53043_solr
   [junit4]   2> 921166 INFO  (zkCallback-2130-thread-1-processing-n:127.0.0.1:35640_solr) [n:127.0.0.1:35640_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 921166 INFO  (zkCallback-2131-thread-1-processing-n:127.0.0.1:60191_solr) [n:127.0.0.1:60191_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 921166 INFO  (zkCallback-2133-thread-1-processing-n:127.0.0.1:53043_solr) [n:127.0.0.1:53043_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 921166 INFO  (zkCallback-2132-thread-1-processing-n:127.0.0.1:55312_solr) [n:127.0.0.1:55312_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 921170 INFO  (zkCallback-2128-thread-1-processing-n:127.0.0.1:38802_solr) [n:127.0.0.1:38802_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 921175 INFO  (zkCallback-2128-thread-1-processing-n:127.0.0.1:38802_solr) [n:127.0.0.1:38802_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 921175 INFO  (zkCallback-2132-thread-1-processing-n:127.0.0.1:55312_solr) [n:127.0.0.1:55312_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 921175 INFO  (zkCallback-2131-thread-1-processing-n:127.0.0.1:60191_solr) [n:127.0.0.1:60191_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 921175 INFO  (zkCallback-2133-thread-1-processing-n:127.0.0.1:53043_solr) [n:127.0.0.1:53043_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 921176 INFO  (zkCallback-2130-thread-1-processing-n:127.0.0.1:35640_solr) [n:127.0.0.1:35640_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 921199 INFO  (jetty-launcher-2106-thread-1) [n:127.0.0.1:35640_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35640_solr
   [junit4]   2> 921200 INFO  (zkCallback-2132-thread-1-processing-n:127.0.0.1:55312_solr) [n:127.0.0.1:55312_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 921200 INFO  (zkCallback-2128-thread-1-processing-n:127.0.0.1:38802_solr) [n:127.0.0.1:38802_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 921200 INFO  (zkCallback-2131-thread-1-processing-n:127.0.0.1:60191_solr) [n:127.0.0.1:60191_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 921200 INFO  (zkCallback-2133-thread-1-processing-n:127.0.0.1:53043_solr) [n:127.0.0.1:53043_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 921200 INFO  (zkCallback-2130-thread-1-processing-n:127.0.0.1:35640_solr) [n:127.0.0.1:35640_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 921259 INFO  (jetty-launcher-2106-thread-5) [n:127.0.0.1:55312_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55312.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 921264 INFO  (jetty-launcher-2106-thread-5) [n:127.0.0.1:55312_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55312.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 921265 INFO  (jetty-launcher-2106-thread-5) [n:127.0.0.1:55312_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55312.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 921266 INFO  (jetty-launcher-2106-thread-5) [n:127.0.0.1:55312_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.rule.RulesTest_6680E89DBC9BEEC3-001/tempDir-001/node5/.
   [junit4]   2> 921282 INFO  (jetty-launcher-2106-thread-1) [n:127.0.0.1:35640_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35640.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 921285 INFO  (jetty-launcher-2106-thread-1) [n:127.0.0.1:35640_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35640.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 921285 INFO  (jetty-launcher-2106-thread-1) [n:127.0.0.1:35640_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35640.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 921299 INFO  (jetty-launcher-2106-thread-2) [n:127.0.0.1:60191_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60191.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 921302 INFO  (jetty-launcher-2106-thread-2) [n:127.0.0.1:60191_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60191.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 921302 INFO  (jetty-launcher-2106-thread-2) [n:127.0.0.1:60191_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60191.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 921303 INFO  (jetty-launcher-2106-thread-2) [n:127.0.0.1:60191_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.rule.RulesTest_6680E89DBC9BEEC3-001/tempDir-001/node1/.
   [junit4]   2> 921304 INFO  (jetty-launcher-2106-thread-1) [n:127.0.0.1:35640_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.rule.RulesTest_6680E89DBC9BEEC3-001/tempDir-001/node2/.
   [junit4]   2> 921342 INFO  (jetty-launcher-2106-thread-3) [n:127.0.0.1:38802_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38802.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 921345 INFO  (jetty-launcher-2106-thread-3) [n:127.0.0.1:38802_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38802.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 921345 INFO  (jetty-launcher-2106-thread-3) [n:127.0.0.1:38802_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38802.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 921346 INFO  (jetty-launcher-2106-thread-3) [n:127.0.0.1:38802_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.rule.RulesTest_6680E89DBC9BEEC3-001/tempDir-001/node3/.
   [junit4]   2> 921411 INFO  (jetty-launcher-2106-thread-4) [n:127.0.0.1:53043_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53043.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 921414 INFO  (jetty-launcher-2106-thread-4) [n:127.0.0.1:53043_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53043.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 921414 INFO  (jetty-launcher-2106-thread-4) [n:127.0.0.1:53043_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53043.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 921415 INFO  (jetty-launcher-2106-thread-4) [n:127.0.0.1:53043_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.rule.RulesTest_6680E89DBC9BEEC3-001/tempDir-001/node4/.
   [junit4]   2> 921470 INFO  (SUITE-RulesTest-seed#[6680E89DBC9BEEC3]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 921471 INFO  (SUITE-RulesTest-seed#[6680E89DBC9BEEC3]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46058/solr ready
   [junit4]   2> 921510 INFO  (TEST-RulesTest.testInvokeApi-seed#[6680E89DBC9BEEC3]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testInvokeApi
   [junit4]   2> 921517 INFO  (qtp983752656-10991) [n:127.0.0.1:60191_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 921518 INFO  (qtp983752656-10991) [n:127.0.0.1:60191_solr    ] o.a.s.h.a.InvokeOp Invocable_invoked {cores=0, freedisk=49}
   [junit4]   2> 921518 INFO  (qtp983752656-10991) [n:127.0.0.1:60191_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/node/invoke params={cores=1&class=org.apache.solr.cloud.rule.ImplicitSnitch&wt=javabin&version=2&freedisk=1} status=0 QTime=1
   [junit4]   2> 921520 INFO  (TEST-RulesTest.testInvokeApi-seed#[6680E89DBC9BEEC3]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 921520 INFO  (TEST-RulesTest.testInvokeApi-seed#[6680E89DBC9BEEC3]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testInvokeApi
   [junit4]   2> 921551 INFO  (TEST-RulesTest.testPortRule-seed#[6680E89DBC9BEEC3]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testPortRule
   [junit4]   2> 921553 INFO  (qtp2092257571-10971) [n:127.0.0.1:38802_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params shards=shard1&replicationFactor=2&collection.configName=conf&snitch=class:ImplicitSnitch&name=portRuleColl&router.name=implicit&nrtReplicas=2&action=CREATE&rule=port:60191&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 921571 INFO  (OverseerThreadFactory-3675-thread-1-processing-n:127.0.0.1:35640_solr) [n:127.0.0.1:35640_solr    ] o.a.s.c.CreateCollectionCmd Create collection portRuleColl
   [junit4]   2> 921697 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=portRuleColl_shard1_replica_n1&action=CREATE&numShards=1&collection=portRuleColl&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 921698 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=portRuleColl_shard1_replica_n2&action=CREATE&numShards=1&collection=portRuleColl&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 921805 INFO  (zkCallback-2131-thread-1-processing-n:127.0.0.1:60191_solr) [n:127.0.0.1:60191_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/portRuleColl/state.json] for collection [portRuleColl] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 922716 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 922719 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 922726 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.s.IndexSchema [portRuleColl_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 922726 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.s.IndexSchema [portRuleColl_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 922728 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 922728 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'portRuleColl_shard1_replica_n1' using configuration from collection portRuleColl, trusted=true
   [junit4]   2> 922728 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 922728 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'portRuleColl_shard1_replica_n2' using configuration from collection portRuleColl, trusted=true
   [junit4]   2> 922728 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60191.solr.core.portRuleColl.shard1.replica_n1' (registry 'solr.core.portRuleColl.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 922728 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 922729 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.c.SolrCore [[portRuleColl_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.rule.RulesTest_6680E89DBC9BEEC3-001/tempDir-001/node1/portRuleColl_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.rule.RulesTest_6680E89DBC9BEEC3-001/tempDir-001/node1/./portRuleColl_shard1_replica_n1/data/]
   [junit4]   2> 922729 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60191.solr.core.portRuleColl.shard1.replica_n2' (registry 'solr.core.portRuleColl.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 922729 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 922729 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.c.SolrCore [[portRuleColl_shard1_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.rule.RulesTest_6680E89DBC9BEEC3-001/tempDir-001/node1/portRuleColl_shard1_replica_n2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.rule.RulesTest_6680E89DBC9BEEC3-001/tempDir-001/node1/./portRuleColl_shard1_replica_n2/data/]
   [junit4]   2> 922778 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 922778 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 922778 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 922778 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 922779 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 922779 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 922779 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 922779 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 922781 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@16020196[portRuleColl_shard1_replica_n1] main]
   [junit4]   2> 922782 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@3b1239f7[portRuleColl_shard1_replica_n2] main]
   [junit4]   2> 922782 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 922782 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 922783 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 922783 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 922784 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570607488621871104
   [junit4]   2> 922786 INFO  (searcherExecutor-3687-thread-1-processing-n:127.0.0.1:60191_solr x:portRuleColl_shard1_replica_n2 s:shard1 c:portRuleColl) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.c.SolrCore [portRuleColl_shard1_replica_n2] Registered new searcher Searcher@3b1239f7[portRuleColl_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 922790 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9998ms
   [junit4]   2> 922790 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 922790 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 922791 INFO  (searcherExecutor-3686-thread-1-processing-n:127.0.0.1:60191_solr x:portRuleColl_shard1_replica_n1 s:shard1 c:portRuleColl) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.c.SolrCore [portRuleColl_shard1_replica_n1] Registered new searcher Searcher@16020196[portRuleColl_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 922791 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570607488629211136
   [junit4]   2> 922891 INFO  (zkCallback-2131-thread-1-processing-n:127.0.0.1:60191_solr) [n:127.0.0.1:60191_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/portRuleColl/state.json] for collection [portRuleColl] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 923290 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 923291 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 923291 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:60191/solr/portRuleColl_shard1_replica_n2/
   [junit4]   2> 923291 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=portRuleColl_shard1_replica_n2 url=http://127.0.0.1:60191/solr START replicas=[http://127.0.0.1:60191/solr/portRuleColl_shard1_replica_n1/] nUpdates=100
   [junit4]   2> 923291 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=portRuleColl_shard1_replica_n2 url=http://127.0.0.1:60191/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 923292 INFO  (qtp983752656-11008) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1 r:core_node1 x:portRuleColl_shard1_replica_n1] o.a.s.c.S.Request [portRuleColl_shard1_replica_n1]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 923293 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 923293 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 923293 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 923296 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:60191/solr/portRuleColl_shard1_replica_n2/ shard1
   [junit4]   2> 923398 INFO  (zkCallback-2131-thread-1-processing-n:127.0.0.1:60191_solr) [n:127.0.0.1:60191_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/portRuleColl/state.json] for collection [portRuleColl] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 923447 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 923449 INFO  (qtp983752656-11006) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=portRuleColl_shard1_replica_n2&action=CREATE&numShards=1&collection=portRuleColl&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1751
   [junit4]   2> 923551 INFO  (zkCallback-2131-thread-1-processing-n:127.0.0.1:60191_solr) [n:127.0.0.1:60191_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/portRuleColl/state.json] for collection [portRuleColl] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 923796 INFO  (qtp983752656-10989) [n:127.0.0.1:60191_solr c:portRuleColl s:shard1  x:portRuleColl_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=portRuleColl_shard1_replica_n1&action=CREATE&numShards=1&collection=portRuleColl&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2099
   [junit4]   2> 923798 INFO  (qtp2092257571-10971) [n:127.0.0.1:38802_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 923897 INFO  (zkCallback-2131-thread-1-processing-n:127.0.0.1:60191_solr) [n:127.0.0.1:60191_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/portRuleColl/state.json] for collection [portRuleColl] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 924799 INFO  (qtp2092257571-10971) [n:127.0.0.1:38802_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={shards=shard1&replicationFactor=2&collection.configName=conf&snitch=class:ImplicitSnitch&name=portRuleColl&router.name=implicit&nrtReplicas=2&action=CREATE&rule=port:60191&wt=javabin&version=2} status=0 QTime=3246
   [junit4]   2> 924801 INFO  (TEST-RulesTest.testPortRule-seed#[6680E89DBC9BEEC3]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 924804 INFO  (qtp2092257571-10973) [n:127.0.0.1:38802_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=portRuleColl&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 924811 INFO  (OverseerCollectionConfigSetProcessor-98162967916445708-127.0.0.1:35640_solr-n_0000000000) [n:127.0.0.1:35640_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 924814 INFO  (OverseerThreadFactory-3675-thread-2-processing-n:127.0.0.1:35640_solr) [n:127.0.0.1:35640_solr    ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 924815 INFO  (qtp983752656-10991) [n:127.0.0.1:60191_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.portRuleColl.shard1.replica_n1, tag=null
   [junit4]   2> 924820 INFO  (qtp983752656-10969) [n:127.0.0.1:60191_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.portRuleColl.shard1.replica_n2, tag=null
   [junit4]   2> 924820 INFO  (qtp983752656-10991) [n:127.0.0.1:60191_solr    ] o.a.s.c.SolrCore [portRuleColl_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4b23c118
   [junit4]   2> 924824 INFO  (qtp983752656-10991) [n:127.0.0.1:60191_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.portRuleColl.shard1.replica_n1, tag=1260634392
   [junit4]   2> 924824 INFO  (qtp983752656-10969) [n:127.0.0.1:60191_solr    ] o.a.s.c.SolrCore [portRuleColl_shard1_replica_n2]  CLOSING SolrCore org.apache.solr.core.SolrCore@275ebf6
   [junit4]   2> 924824 INFO  (qtp983752656-10991) [n:127.0.0.1:60191_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.portRuleColl.shard1.leader, tag=1260634392
   [junit4]   2> 924825 INFO  (qtp983752656-10991) [n:127.0.0.1:60191_solr    ] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration.
   [junit4]   2> 924826 INFO  (qtp983752656-10991) [n:127.0.0.1:60191_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=portRuleColl_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=11
   [junit4]   2> 924827 INFO  (qtp983752656-10969) [n:127.0.0.1:60191_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.portRuleColl.shard1.replica_n2, tag=41282550
   [junit4]   2> 924827 INFO  (qtp983752656-10969) [n:127.0.0.1:60191_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.portRuleColl.shard1.leader, tag=41282550
   [junit4]   2> 924851 WARN  (zkCallback-2131-thread-1-processing-n:127.0.0.1:60191_solr) [n:127.0.0.1:60191_solr    ] o.a.s.c.LeaderElector Our node is no longer in line to be leader
   [junit4]   2> 924854 INFO  (qtp983752656-10969) [n:127.0.0.1:60191_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=portRuleColl_shard1_replica_n2&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=35
   [junit4]   2> 925558 INFO  (qtp2092257571-10973) [n:127.0.0.1:38802_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=portRuleColl&action=DELETE&wt=javabin&version=2} status=0 QTime=753
   [junit4]   2> 925558 INFO  (TEST-RulesTest.testPortRule-seed#[6680E89DBC9BEEC3]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testPortRule
   [junit4]   2> 925587 INFO  (TEST-RulesTest.testHostFragmentRuleThrowsExceptionWhenIpDoesNotMatch-seed#[6680E89DBC9BEEC3]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testHostFragmentRuleThrowsExceptionWhenIpDoesNotMatch
   [junit4]   2> 925595 INFO  (qtp2092257571-10974) [n:127.0.0.1:38802_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params shards=shard1&replicationFactor=2&collection.configName=conf&snitch=class:ImplicitSnitch&name=ipRuleColl&router.name=implicit&nrtReplicas=2&action=CREATE&rule=ip_2:0&rule=ip_1:19999&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 925597 INFO  (OverseerCollectionConfigSetProcessor-98162967916445708-127.0.0.1:35640_solr-n_0000000000) [n:127.0.0.1:35640_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 925598 INFO  (OverseerThreadFactory-3675-thread-3-processing-n:127.0.0.1:35640_solr) [n:127.0.0.1:35640_solr    ] o.a.s.c.CreateCollectionCmd Create collection ipRuleColl
   [junit4]   2> 925600 ERROR (OverseerThreadFactory-3675-thread-3-processing-n:127.0.0.1:35640_solr) [n:127.0.0.1:35640_solr    ] o.a.s.c.OverseerCollectionMessageHandler Collection: ipRuleColl operation: create failed:org.apache.solr.common.SolrException: Could not identify nodes matching the rules [{"ip_2":"0"}, {"ip_1":"19999"}]
   [junit4]   2>  tag values{
   [junit4]   2>   "127.0.0.1:38802_solr":{
   [junit4]   2>     "ip_1":"1",
   [junit4]   2>     "ip_2":"0"},
   [junit4]   2>   "127.0.0.1:60191_solr":{
   [junit4]   2>     "ip_1":"1",
   [junit4]   2>     "ip_2":"0"},
   [junit4]   2>   "127.0.0.1:55312_solr":{
   [junit4]   2>     "ip_1":"1",
   [junit4]   2>     "ip_2":"0"},
   [junit4]   2>   "127.0.0.1:35640_solr":{
   [junit4]   2>     "ip_1":"1",
   [junit4]   2>     "ip_2":"0"},
   [junit4]   2>   "127.0.0.1:53043_solr":{
   [junit4]   2>     "ip_1":"1",
   [junit4]   2>     "ip_2":"0"}}
   [junit4]   2> at org.apache.solr.cloud.rule.ReplicaAssigner.getNodeMappings(ReplicaAssigner.java:146)
   [junit4]   2> at org.apache.solr.cloud.OverseerCollectionMessageHandler.identifyNodes(OverseerCollectionMessageHandler.java:763)
   [junit4]   2> at org.apache.solr.cloud.CreateCollectionCmd.call(CreateCollectionCmd.java:165)
   [junit4]   2> at org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:233)
   [junit4]   2> at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:464)
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 925601 INFO  (qtp2092257571-10974) [n:127.0.0.1:38802_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={shards=shard1&replicationFactor=2&collection.configName=conf&snitch=class:ImplicitSnitch&name=ipRuleColl&router.name=implicit&nrtReplicas=2&action=CREATE&rule=ip_2:0&rule=ip_1:19999&wt=javabin&version=2} status=400 QTime=6
   [junit4]   2> 925603 INFO  (TEST-RulesTest.testHostFragmentRuleThrowsExceptionWhenIpDoesNotMatch-seed#[6680E89DBC9BEEC3]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 925604 INFO  (TEST-RulesTest.testHostFragmentRuleThrowsExceptionWhenIpDoesNotMatch-seed#[6680E89DBC9BEEC3]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testHostFragmentRuleThrowsExceptionWhenIpDoesNotMatch
   [junit4]   2> 925632 INFO  (TEST-RulesTest.doIntegrationTest-seed#[6680E89DBC9BEEC3]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doIntegrationTest
   [junit4]   2> 925633 INFO  (qtp2092257571-10970) [n:127.0.0.1:38802_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params shards=shard1&replicationFactor=2&collection.configName=conf&snitch=class:ImplicitSnitch&name=rulesColl&router.name=implicit&nrtReplicas=2&action=CREATE&rule=cores:<4&rule=node:*,replica:<2&rule=freedisk:>0&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 925634 INFO  (OverseerThreadFactory-3675-thread-4-processing-n:127.0.0.1:35640_solr) [n:127.0.0.1:35640_solr    ] o.a.s.c.CreateCollectionCmd Create collection rulesColl
   [junit4]   2> 925634 INFO  (OverseerCollectionConfigSetProcessor-98162967916445708-127.0.0.1:35640_solr-n_0000000000) [n:127.0.0.1:35640_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 925636 INFO  (qtp1597722424-10982) [n:127.0.0.1:55312_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 925636 INFO  (qtp1597722424-10982) [n:127.0.0.1:55312_solr    ] o.a.s.h.a.InvokeOp Invocable_invoked {cores=0, freedisk=49}
   [junit4]   2> 925636 INFO  (qtp1597722424-10982) [n:127.0.0.1:55312_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={cores=1&action=INVOKE&class=org.apache.solr.cloud.rule.ImplicitSnitch&wt=javabin&version=2&freedisk=1} status=0 QTime=0
   [junit4]   2> 925637 INFO  (qtp1515529463-10977) [n:127.0.0.1:35640_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 925637 INFO  (qtp1515529463-10977) [n:127.0.0.1:35640_solr    ] o.a.s.h.a.InvokeOp Invocable_invoked {cores=0, freedisk=49}
   [junit4]   2> 925637 INFO  (qtp1515529463-10977) [n:127.0.0.1:35640_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={cores=1&action=INVOKE&class=org.apache.solr.cloud.rule.ImplicitSnitch&wt=javabin&version=2&freedisk=1} status=0 QTime=0
   [junit4]   2> 925638 INFO  (qtp983752656-11008) [n:127.0.0.1:60191_solr    ] o.a.s.h.a.InvokeOp Invocable_invoked {cores=0, freedisk=49}
   [junit4]   2> 925638 INFO  (qtp983752656-11008) [n:127.0.0.1:60191_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={cores=1&action=INVOKE&class=org.apache.solr.cloud.rule.ImplicitSnitch&wt=javabin&version=2&freedisk=1} status=0 QTime=0
   [junit4]   2> 925639 INFO  (qtp1525702218-10994) [n:127.0.0.1:53043_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 925639 INFO  (qtp1525702218-10994) [n:127.0.0.1:53043_solr    ] o.a.s.h.a.InvokeOp Invocable_invoked {cores=0, freedisk=49}
   [junit4]   2> 925639 INFO  (qtp1525702218-10994) [n:127.0.0.1:53043_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={cores=1&action=INVOKE&class=org.apache.solr.cloud.rule.ImplicitSnitch&wt=javabin&version=2&freedisk=1} status=0 QTime=0
   [junit4]   2> 925640 INFO  (qtp2092257571-10964) [n:127.0.0.1:38802_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 925640 INFO  (qtp2092257571-10964) [n:127.0.0.1:38802_solr    ] o.a.s.h.a.InvokeOp Invocable_invoked {cores=0, freedisk=49}
   [junit4]   2> 925640 INFO  (qtp2092257571-10964) [n:127.0.0.1:38802_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={cores=1&action=INVOKE&class=org.apache.solr.cloud.rule.ImplicitSnitch&wt=javabin&version=2&freedisk=1} status=0 QTime=0
   [junit4]   2> 925845 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=rulesColl_shard1_replica_n1&action=CREATE&numShards=1&collection=rulesColl&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 925845 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=rulesColl_shard1_replica_n2&action=CREATE&numShards=1&collection=rulesColl&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 925955 INFO  (zkCallback-2130-thread-1-processing-n:127.0.0.1:35640_solr) [n:127.0.0.1:35640_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/rulesColl/state.json] for collection [rulesColl] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 925955 INFO  (zkCallback-2132-thread-1-processing-n:127.0.0.1:55312_solr) [n:127.0.0.1:55312_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/rulesColl/state.json] for collection [rulesColl] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 926856 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 926862 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.s.IndexSchema [rulesColl_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 926864 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 926864 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 926864 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'rulesColl_shard1_replica_n2' using configuration from collection rulesColl, trusted=true
   [junit4]   2> 926864 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35640.solr.core.rulesColl.shard1.replica_n2' (registry 'solr.core.rulesColl.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 926865 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 926865 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.c.SolrCore [[rulesColl_shard1_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.rule.RulesTest_6680E89DBC9BEEC3-001/tempDir-001/node2/rulesColl_shard1_replica_n2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.rule.RulesTest_6680E89DBC9BEEC3-001/tempDir-001/node2/./rulesColl_shard1_replica_n2/data/]
   [junit4]   2> 926871 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.s.IndexSchema [rulesColl_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 926873 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 926873 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'rulesColl_shard1_replica_n1' using configuration from collection rulesColl, trusted=true
   [junit4]   2> 926873 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55312.solr.core.rulesColl.shard1.replica_n1' (registry 'solr.core.rulesColl.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c16bd16
   [junit4]   2> 926874 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 926874 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.c.SolrCore [[rulesColl_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.rule.RulesTest_6680E89DBC9BEEC3-001/tempDir-001/node5/rulesColl_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.rule.RulesTest_6680E89DBC9BEEC3-001/tempDir-001/node5/./rulesColl_shard1_replica_n1/data/]
   [junit4]   2> 926961 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 926961 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 926962 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 926962 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 926964 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@342d19eb[rulesColl_shard1_replica_n2] main]
   [junit4]   2> 926965 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 926965 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 926965 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 926965 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 926966 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 926966 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570607493007015936
   [junit4]   2> 926966 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 926966 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 926968 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@16c72afa[rulesColl_shard1_replica_n1] main]
   [junit4]   2> 926969 INFO  (searcherExecutor-3696-thread-1-processing-n:127.0.0.1:35640_solr x:rulesColl_shard1_replica_n2 s:shard1 c:rulesColl) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.c.SolrCore [rulesColl_shard1_replica_n2] Registered new searcher Searcher@342d19eb[rulesColl_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 926972 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 926973 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 926973 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 926973 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 926974 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570607493015404544
   [junit4]   2> 926977 INFO  (searcherExecutor-3697-thread-1-processing-n:127.0.0.1:55312_solr x:rulesColl_shard1_replica_n1 s:shard1 c:rulesColl) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.c.SolrCore [rulesColl_shard1_replica_n1] Registered new searcher Searcher@16c72afa[rulesColl_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 927073 INFO  (zkCallback-2130-thread-1-processing-n:127.0.0.1:35640_solr) [n:127.0.0.1:35640_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/rulesColl/state.json] for collection [rulesColl] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 927073 INFO  (zkCallback-2132-thread-1-processing-n:127.0.0.1:55312_solr) [n:127.0.0.1:55312_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/rulesColl/state.json] for collection [rulesColl] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 927472 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 927473 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 927473 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:35640/solr/rulesColl_shard1_replica_n2/
   [junit4]   2> 927473 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=rulesColl_shard1_replica_n2 url=http://127.0.0.1:35640/solr START replicas=[http://127.0.0.1:55312/solr/rulesColl_shard1_replica_n1/] nUpdates=100
   [junit4]   2> 927475 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=rulesColl_shard1_replica_n2 url=http://127.0.0.1:35640/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 927476 INFO  (qtp1597722424-10999) [n:127.0.0.1:55312_solr c:rulesColl s:shard1 r:core_node2 x:rulesColl_shard1_replica_n1] o.a.s.c.S.Request [rulesColl_shard1_replica_n1]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 927477 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 927477 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 927477 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 927479 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:35640/solr/rulesColl_shard1_replica_n2/ shard1
   [junit4]   2> 927580 INFO  (zkCallback-2130-thread-1-processing-n:127.0.0.1:35640_solr) [n:127.0.0.1:35640_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/rulesColl/state.json] for collection [rulesColl] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 927580 INFO  (zkCallback-2132-thread-1-processing-n:127.0.0.1:55312_solr) [n:127.0.0.1:55312_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/rulesColl/state.json] for collection [rulesColl] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 927631 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 927635 INFO  (qtp1515529463-10985) [n:127.0.0.1:35640_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=rulesColl_shard1_replica_n2&action=CREATE&numShards=1&collection=rulesColl&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1789
   [junit4]   2> 927738 INFO  (zkCallback-2130-thread-1-processing-n:127.0.0.1:35640_solr) [n:127.0.0.1:35640_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/rulesColl/state.json] for collection [rulesColl] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 927738 INFO  (zkCallback-2132-thread-1-processing-n:127.0.0.1:55312_solr) [n:127.0.0.1:55312_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/rulesColl/state.json] for collection [rulesColl] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 927986 INFO  (qtp1597722424-11000) [n:127.0.0.1:55312_solr c:rulesColl s:shard1  x:rulesColl_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=rulesColl_shard1_replica_n1&action=CREATE&numShards=1&collection=rulesColl&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2141
   [junit4]   2> 927988 INFO  (qtp2092257571-10970) [n:127.0.0.1:38802_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 928089 INFO  (zkCallback-2132-thread-1-processing-n:127.0.0.1:55312_solr) [n:127.0.0.1:55312_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/rulesColl/state.json] for collection [rulesColl] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 928089 INFO  (zkCallback-2130-thread-1-processing-n:127.0.0.1:35640_solr) [n:127.0.0.1:35640_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/rulesColl/state.json] for collection [rulesColl] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 928989 INFO  (qtp2092257571-10970) [n:127.0.0.1:38802_solr    ] o.a.s.s.HttpSolrCall [

[...truncated too long message...]

r c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=1361890778
   [junit4]   2> 191945 INFO  (coreCloseExecutor-692-thread-1) [n:127.0.0.1:52134_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard2.leader, tag=846098925
   [junit4]   2> 191946 INFO  (jetty-closer-444-thread-4) [    ] o.a.s.c.Overseer Overseer (id=98163170569945098-127.0.0.1:35684_solr-n_0000000000) closing
   [junit4]   2> 191947 INFO  (OverseerStateUpdate-98163170569945098-127.0.0.1:35684_solr-n_0000000000) [n:127.0.0.1:35684_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:35684_solr
   [junit4]   2> 191947 INFO  (zkCallback-461-thread-3-processing-n:127.0.0.1:35684_solr) [n:127.0.0.1:35684_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 193427 WARN  (zkCallback-465-thread-2-processing-n:127.0.0.1:34427_solr) [n:127.0.0.1:34427_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 193427 INFO  (jetty-closer-444-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6bfe15b0{/solr,null,UNAVAILABLE}
   [junit4]   2> 193447 WARN  (zkCallback-463-thread-1-processing-n:127.0.0.1:52134_solr) [n:127.0.0.1:52134_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 193448 INFO  (jetty-closer-444-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3b360657{/solr,null,UNAVAILABLE}
   [junit4]   2> 193449 WARN  (zkCallback-461-thread-3-processing-n:127.0.0.1:35684_solr) [n:127.0.0.1:35684_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 193449 INFO  (jetty-closer-444-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@28f82b85{/solr,null,UNAVAILABLE}
   [junit4]   2> 193450 ERROR (SUITE-StreamExpressionTest-seed#[29C234860309F872]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 193450 INFO  (SUITE-StreamExpressionTest-seed#[29C234860309F872]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:55824 55824
   [junit4]   2> 193552 INFO  (Thread-174) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:55824 55824
   [junit4]   2> 193553 WARN  (Thread-174) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 91 /solr/aliases.json
   [junit4]   2> 13 /solr/configs/conf
   [junit4]   2> 8 /solr/configs/ml
   [junit4]   2> 4 /solr/security.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 91 /solr/clusterprops.json
   [junit4]   2> 91 /solr/clusterstate.json
   [junit4]   2> 8 /solr/collections/destinationCollection/state.json
   [junit4]   2> 4 /solr/collections/destination/state.json
   [junit4]   2> 4 /solr/collections/parallelDestinationCollection/state.json
   [junit4]   2> 4 /solr/collections/parallelDestinationCollection1/state.json
   [junit4]   2> 4 /solr/collections/mainCorpus/state.json
   [junit4]   2> 2 /solr/overseer_elect/election/98163170569945097-127.0.0.1:52134_solr-n_0000000001
   [junit4]   2> 2 /solr/collections/collection1/state.json
   [junit4]   2> 2 /solr/collections/workQueue/state.json
   [junit4]   2> 2 /solr/collections/uknownCollection/state.json
   [junit4]   2> 2 /solr/collections/modelCollection/state.json
   [junit4]   2> 2 /solr/collections/checkpointCollection/state.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 91 /solr/live_nodes
   [junit4]   2> 91 /solr/collections
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_29C234860309F872-001
   [junit4]   2> Jun 19, 2017 5:47:00 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {name_s=PostingsFormat(name=LuceneFixedGap), expr_s=FSTOrd50, iteration_i=FSTOrd50, terms_ss=PostingsFormat(name=LuceneVarGapDocFreqInterval), join2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), multiDefault=PostingsFormat(name=LuceneVarGapDocFreqInterval), subject=PostingsFormat(name=LuceneVarGapDocFreqInterval), a_f=PostingsFormat(name=LuceneVarGapDocFreqInterval), intDefault=PostingsFormat(name=Asserting), a2_s=PostingsFormat(name=LuceneFixedGap), falseNegative_i=PostingsFormat(name=Asserting), a_i=FSTOrd50, level1_s=FSTOrd50, body_t=PostingsFormat(name=LuceneFixedGap), a_s=PostingsFormat(name=LuceneFixedGap), a_t=PostingsFormat(name=Asserting), id=FSTOrd50, text=PostingsFormat(name=LuceneFixedGap), alpha_d=PostingsFormat(name=LuceneFixedGap), i_multi=FSTOrd50, idf_d=PostingsFormat(name=LuceneVarGapDocFreqInterval), b_ls=PostingsFormat(name=LuceneVarGapDocFreqInterval), checkpoint_ss=PostingsFormat(name=LuceneFixedGap), test_t=PostingsFormat(name=LuceneFixedGap), ident_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), order_i=PostingsFormat(name=LuceneVarGapDocFreqInterval), error_d=FSTOrd50, side_s=PostingsFormat(name=LuceneFixedGap), truePositive_i=FSTOrd50, miles_i=PostingsFormat(name=LuceneVarGapDocFreqInterval), field_s=PostingsFormat(name=Asserting), price_f=PostingsFormat(name=LuceneVarGapDocFreqInterval), a1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), join1_i=FSTOrd50, join3_i=PostingsFormat(name=LuceneFixedGap), test_i=PostingsFormat(name=LuceneVarGapDocFreqInterval), falsePositive_i=PostingsFormat(name=Asserting), field_i=PostingsFormat(name=LuceneVarGapDocFreqInterval), s_multi=PostingsFormat(name=LuceneFixedGap), a_ss=PostingsFormat(name=Asserting), col_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), level2_s=PostingsFormat(name=Asserting), score_f=PostingsFormat(name=LuceneFixedGap), tv_text=PostingsFormat(name=Asserting), term_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), text_s=FSTOrd50, timestamp=PostingsFormat(name=Asserting), idfs_ds=PostingsFormat(name=Asserting), weights_ds=PostingsFormat(name=LuceneFixedGap), featureSet_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), trueNegative_i=PostingsFormat(name=LuceneFixedGap), index_i=PostingsFormat(name=LuceneVarGapDocFreqInterval), test_dt=FSTOrd50, out_i=PostingsFormat(name=LuceneVarGapDocFreqInterval), whitetok=PostingsFormat(name=LuceneFixedGap), _version_=PostingsFormat(name=Asserting)}, docValues:{name_s=DocValuesFormat(name=Direct), expr_s=DocValuesFormat(name=Asserting), iteration_i=DocValuesFormat(name=Asserting), terms_ss=DocValuesFormat(name=Lucene70), join2_s=DocValuesFormat(name=Lucene70), field_s=DocValuesFormat(name=Lucene70), multiDefault=DocValuesFormat(name=Lucene70), price_f=DocValuesFormat(name=Lucene70), a1_s=DocValuesFormat(name=Lucene70), a_f=DocValuesFormat(name=Lucene70), intDefault=DocValuesFormat(name=Lucene70), join1_i=DocValuesFormat(name=Asserting), a2_s=DocValuesFormat(name=Direct), falseNegative_i=DocValuesFormat(name=Lucene70), a_i=DocValuesFormat(name=Asserting), join3_i=DocValuesFormat(name=Direct), test_i=DocValuesFormat(name=Lucene70), falsePositive_i=DocValuesFormat(name=Lucene70), field_i=DocValuesFormat(name=Lucene70), level1_s=DocValuesFormat(name=Asserting), s_multi=DocValuesFormat(name=Direct), a_ss=DocValuesFormat(name=Lucene70), col_s=DocValuesFormat(name=Lucene70), level2_s=DocValuesFormat(name=Lucene70), a_s=DocValuesFormat(name=Direct), score_f=DocValuesFormat(name=Direct), term_s=DocValuesFormat(name=Lucene70), text_s=DocValuesFormat(name=Asserting), id=DocValuesFormat(name=Asserting), timestamp=DocValuesFormat(name=Lucene70), i_multi=DocValuesFormat(name=Asserting), featureSet_s=DocValuesFormat(name=Lucene70), trueNegative_i=DocValuesFormat(name=Direct), index_i=DocValuesFormat(name=Lucene70), b_ls=DocValuesFormat(name=Lucene70), test_dt=DocValuesFormat(name=Asserting), out_i=DocValuesFormat(name=Lucene70), checkpoint_ss=DocValuesFormat(name=Direct), ident_s=DocValuesFormat(name=Lucene70), order_i=DocValuesFormat(name=Lucene70), _version_=DocValuesFormat(name=Lucene70), side_s=DocValuesFormat(name=Direct), truePositive_i=DocValuesFormat(name=Asserting), miles_i=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1915, maxMBSortInHeap=5.550173099092766, sim=RandomSimilarity(queryNorm=false): {}, locale=es-HN, timezone=Etc/GMT+7
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=220869136,total=472383488
   [junit4]   2> NOTE: All tests run in this JVM: [HttpSolrClientSSLAuthConPoolTest, SolrExampleBinaryTest, JsonValidatorTest, GraphTest, ConcurrentUpdateSolrClientTest, SolrSchemalessExampleTest, JettyWebappTest, GreaterThanEqualToEvaluatorTest, DocumentAnalysisResponseTest, SolrExampleJettyTest, CubedRootEvaluatorTest, NoOpResponseParserTest, RoundEvaluatorTest, ModifiableSolrParamsTest, CommonAdminParamsTest, TestCollectionAdminRequest, MultiplyEvaluatorTest, TestJavaBinCodec, LargeVolumeJettyTest, SolrExampleXMLTest, SolrParamTest, TestCoreAdmin, BasicHttpSolrClientTest, HttpSolrClientBuilderTest, JdbcTest, StreamExpressionTest]
   [junit4] Completed [135/135 (1!)] on J2 in 107.64s, 89 tests, 1 error <<< FAILURES!

[...truncated 42591 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1331 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1331/

No tests ran.

Build Log:
[...truncated 22 lines...]
ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/test-data
org.tmatesoft.svn.core.SVNAuthenticationException: svn: E170001: OPTIONS of '/repos/asf/lucene/test-data': 403 Forbidden (http://svn.apache.org)
        at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:68)
        at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:798)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:391)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:379)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:862)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:698)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:118)
        at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1049)
        at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:149)
        at org.tmatesoft.svn.core.internal.wc2.SvnRepositoryAccess.createRepository(SvnRepositoryAccess.java:121)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgRepositoryAccess.createRepository(SvnNgRepositoryAccess.java:211)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:194)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:111)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:40)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:18)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
        at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
        at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1235)
        at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
        at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:158)
        at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:162)
        at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:996)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:972)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:948)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731)
        at hudson.remoting.UserRequest.perform(UserRequest.java:153)
        at hudson.remoting.UserRequest.perform(UserRequest.java:50)
        at hudson.remoting.Request$2.run(Request.java:336)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
ERROR: Subversion update failed
hudson.remoting.ProxyException: org.tmatesoft.svn.core.SVNAuthenticationException: svn: E170001: OPTIONS of '/repos/asf/lucene/test-data': 403 Forbidden (http://svn.apache.org)
        at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:68)
        at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:798)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:391)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:379)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:862)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:698)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:118)
        at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1049)
        at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:149)
        at org.tmatesoft.svn.core.internal.wc2.SvnRepositoryAccess.createRepository(SvnRepositoryAccess.java:121)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgRepositoryAccess.createRepository(SvnNgRepositoryAccess.java:211)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:194)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:111)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:40)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:18)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
        at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
        at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1235)
        at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
        at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:158)
Caused: hudson.remoting.ProxyException: hudson.scm.subversion.UpdaterException: failed to perform svn update
        at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:212)
        at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:162)
        at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:996)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:972)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:948)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731)
        at hudson.remoting.UserRequest.perform(UserRequest.java:153)
        at hudson.remoting.UserRequest.perform(UserRequest.java:50)
        at hudson.remoting.Request$2.run(Request.java:336)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused: hudson.remoting.ProxyException: java.io.IOException
        at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:212)
        at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:162)
        at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:996)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:972)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:948)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731)
        at hudson.remoting.UserRequest.perform(UserRequest.java:153)
        at hudson.remoting.UserRequest.perform(UserRequest.java:50)
        at hudson.remoting.Request$2.run(Request.java:336)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
        at ......remote call to lucene(Native Method)
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
        at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
        at hudson.remoting.Channel.call(Channel.java:830)
        at hudson.FilePath.act(FilePath.java:985)
Caused: java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master at hudson.remoting.Channel@40899b59:lucene
        at hudson.FilePath.act(FilePath.java:992)
        at hudson.FilePath.act(FilePath.java:974)
        at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:897)
        at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:833)
        at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
        at hudson.scm.SCM.checkout(SCM.java:496)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1281)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
        at hudson.model.Run.execute(Run.java:1728)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:98)
        at hudson.model.Executor.run(Executor.java:405)
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git://git.apache.org/lucene-solr.git # timeout=10
Fetching upstream changes from git://git.apache.org/lucene-solr.git
 > git --version # timeout=10
 > git fetch --tags --progress git://git.apache.org/lucene-solr.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 3ff8c870243c576f73650309715f6cbd3bc8c869 (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 3ff8c870243c576f73650309715f6cbd3bc8c869
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Cleaning up /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data
Updating http://svn.apache.org/repos/asf/lucene/test-data at revision '2017-06-20T01:37:47.427 +0000'
ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/test-data
org.tmatesoft.svn.core.SVNAuthenticationException: svn: E170001: OPTIONS of '/repos/asf/lucene/test-data': 403 Forbidden (http://svn.apache.org)
        at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:68)
        at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:798)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:391)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:379)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:862)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:698)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:118)
        at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1049)
        at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:149)
        at org.tmatesoft.svn.core.internal.wc2.SvnRepositoryAccess.createRepository(SvnRepositoryAccess.java:121)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgRepositoryAccess.createRepository(SvnNgRepositoryAccess.java:211)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:194)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:111)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:40)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:18)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
        at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
        at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1235)
        at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
        at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:158)
        at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:162)
        at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:996)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:972)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:948)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731)
        at hudson.remoting.UserRequest.perform(UserRequest.java:153)
        at hudson.remoting.UserRequest.perform(UserRequest.java:50)
        at hudson.remoting.Request$2.run(Request.java:336)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
ERROR: Subversion update failed
hudson.remoting.ProxyException: org.tmatesoft.svn.core.SVNAuthenticationException: svn: E170001: OPTIONS of '/repos/asf/lucene/test-data': 403 Forbidden (http://svn.apache.org)
        at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:68)
        at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:798)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:391)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:379)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:862)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:698)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:118)
        at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1049)
        at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:149)
        at org.tmatesoft.svn.core.internal.wc2.SvnRepositoryAccess.createRepository(SvnRepositoryAccess.java:121)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgRepositoryAccess.createRepository(SvnNgRepositoryAccess.java:211)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:194)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:111)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:40)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:18)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
        at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
        at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1235)
        at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
        at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:158)
Caused: hudson.remoting.ProxyException: hudson.scm.subversion.UpdaterException: failed to perform svn update
        at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:212)
        at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:162)
        at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:996)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:972)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:948)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731)
        at hudson.remoting.UserRequest.perform(UserRequest.java:153)
        at hudson.remoting.UserRequest.perform(UserRequest.java:50)
        at hudson.remoting.Request$2.run(Request.java:336)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused: hudson.remoting.ProxyException: java.io.IOException
        at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:212)
        at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:162)
        at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:996)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:972)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:948)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731)
        at hudson.remoting.UserRequest.perform(UserRequest.java:153)
        at hudson.remoting.UserRequest.perform(UserRequest.java:50)
        at hudson.remoting.Request$2.run(Request.java:336)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
        at ......remote call to lucene(Native Method)
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
        at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
        at hudson.remoting.Channel.call(Channel.java:830)
        at hudson.FilePath.act(FilePath.java:985)
Caused: java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master at hudson.remoting.Channel@40899b59:lucene
        at hudson.FilePath.act(FilePath.java:992)
        at hudson.FilePath.act(FilePath.java:974)
        at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:897)
        at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:833)
        at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
        at hudson.scm.SCM.checkout(SCM.java:496)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1281)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
        at hudson.model.Run.execute(Run.java:1728)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:98)
        at hudson.model.Executor.run(Executor.java:405)
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git://git.apache.org/lucene-solr.git # timeout=10
Fetching upstream changes from git://git.apache.org/lucene-solr.git
 > git --version # timeout=10
 > git fetch --tags --progress git://git.apache.org/lucene-solr.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 3ff8c870243c576f73650309715f6cbd3bc8c869 (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 3ff8c870243c576f73650309715f6cbd3bc8c869
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Cleaning up /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data
Updating http://svn.apache.org/repos/asf/lucene/test-data at revision '2017-06-20T01:37:47.427 +0000'
ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/test-data
org.tmatesoft.svn.core.SVNAuthenticationException: svn: E170001: OPTIONS of '/repos/asf/lucene/test-data': 403 Forbidden (http://svn.apache.org)
        at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:68)
        at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:798)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:391)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:379)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:862)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:698)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:118)
        at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1049)
        at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:149)
        at org.tmatesoft.svn.core.internal.wc2.SvnRepositoryAccess.createRepository(SvnRepositoryAccess.java:121)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgRepositoryAccess.createRepository(SvnNgRepositoryAccess.java:211)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:194)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:111)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:40)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:18)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
        at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
        at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1235)
        at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
        at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:158)
        at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:162)
        at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:996)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:972)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:948)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731)
        at hudson.remoting.UserRequest.perform(UserRequest.java:153)
        at hudson.remoting.UserRequest.perform(UserRequest.java:50)
        at hudson.remoting.Request$2.run(Request.java:336)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
ERROR: Subversion update failed
hudson.remoting.ProxyException: org.tmatesoft.svn.core.SVNAuthenticationException: svn: E170001: OPTIONS of '/repos/asf/lucene/test-data': 403 Forbidden (http://svn.apache.org)
        at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:68)
        at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:798)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:391)
        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:379)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:862)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:698)
        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:118)
        at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1049)
        at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:149)
        at org.tmatesoft.svn.core.internal.wc2.SvnRepositoryAccess.createRepository(SvnRepositoryAccess.java:121)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgRepositoryAccess.createRepository(SvnNgRepositoryAccess.java:211)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:194)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:111)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:40)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:18)
        at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
        at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
        at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1235)
        at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
        at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:158)
Caused: hudson.remoting.ProxyException: hudson.scm.subversion.UpdaterException: failed to perform svn update
        at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:212)
        at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:162)
        at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:996)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:972)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:948)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731)
        at hudson.remoting.UserRequest.perform(UserRequest.java:153)
        at hudson.remoting.UserRequest.perform(UserRequest.java:50)
        at hudson.remoting.Request$2.run(Request.java:336)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused: hudson.remoting.ProxyException: java.io.IOException
        at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:212)
        at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:162)
        at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:996)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:972)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:948)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731)
        at hudson.remoting.UserRequest.perform(UserRequest.java:153)
        at hudson.remoting.UserRequest.perform(UserRequest.java:50)
        at hudson.remoting.Request$2.run(Request.java:336)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
        at ......remote call to lucene(Native Method)
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
        at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
        at hudson.remoting.Channel.call(Channel.java:830)
        at hudson.FilePath.act(FilePath.java:985)
Caused: java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master at hudson.remoting.Channel@40899b59:lucene
        at hudson.FilePath.act(FilePath.java:992)
        at hudson.FilePath.act(FilePath.java:974)
        at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:897)
        at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:833)
        at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
        at hudson.scm.SCM.checkout(SCM.java:496)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1281)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
        at hudson.model.Run.execute(Run.java:1728)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:98)
        at hudson.model.Executor.run(Executor.java:405)
Archiving artifacts
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1332 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1332/

2 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest.testPolicyAndPreferences

Error Message:
Adding a policy with 'cores' attribute should not have succeeded.

Stack Trace:
java.lang.AssertionError: Adding a policy with 'cores' attribute should not have succeeded.
        at __randomizedtesting.SeedInfo.seed([389D3A4D00E5B7D1:9955646479A3B1DF]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest.testPolicyAndPreferences(AutoScalingHandlerTest.java:83)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.update.HardAutoCommitTest.testCommitWithin

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
        at __randomizedtesting.SeedInfo.seed([389D3A4D00E5B7D1:824F553583CB59C4]:0)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:890)
        at org.apache.solr.update.HardAutoCommitTest.testCommitWithin(HardAutoCommitTest.java:100)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
        xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int></lst><result name="response" numFound="0" start="0"></result>
</response>

        request was:q=id:529&qt=standard&start=0&rows=20&version=2.2
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:883)
        ... 40 more




Build Log:
[...truncated 11729 lines...]
   [junit4] Suite: org.apache.solr.update.HardAutoCommitTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.update.HardAutoCommitTest_389D3A4D00E5B7D1-001/init-core-data-001
   [junit4]   2> 461419 WARN  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 461424 INFO  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 461425 INFO  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 461427 INFO  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 461427 INFO  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 461471 INFO  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 461520 INFO  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 461726 INFO  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 461813 INFO  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c76aaa2
   [junit4]   2> 461825 INFO  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c76aaa2
   [junit4]   2> 461826 INFO  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c76aaa2
   [junit4]   2> 461839 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 461902 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 461950 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 462087 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 462094 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 462094 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c76aaa2
   [junit4]   2> 462094 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 462094 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.update.HardAutoCommitTest_389D3A4D00E5B7D1-001/init-core-data-001/]
   [junit4]   2> 462096 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=91.7294921875, floorSegmentMB=0.8935546875, forceMergeDeletesPctAllowed=8.600203229093585, segmentsPerTier=44.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7878558064007951
   [junit4]   2> 462187 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 462187 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 462188 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 462188 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 462189 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1219746815, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 462190 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@73ee5e5c[collection1] main]
   [junit4]   2> 462191 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 462202 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 462271 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 462271 INFO  (searcherExecutor-1173-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 462271 INFO  (coreLoadExecutor-1172-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570792280886870016
   [junit4]   2> 462271 INFO  (searcherExecutor-1173-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 462271 INFO  (searcherExecutor-1173-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 462271 INFO  (searcherExecutor-1173-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 462272 INFO  (searcherExecutor-1173-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 462272 INFO  (searcherExecutor-1173-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 462272 INFO  (searcherExecutor-1173-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 462272 INFO  (searcherExecutor-1173-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 462273 INFO  (searcherExecutor-1173-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 462275 INFO  (searcherExecutor-1173-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@73ee5e5c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 462277 INFO  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 462279 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCommitWithin
   [junit4]   2> 462279 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 462282 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@d822769[collection1] realtime]
   [junit4]   2> 462282 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:* (-9223372036854775807)} 0 3
   [junit4]   2> 462283 ERROR (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    ] o.a.s.c.CorePropertiesLocator Couldn't load core descriptor from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/core.properties:java.nio.file.NoSuchFileException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/core.properties
   [junit4]   2> 462284 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 462419 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 462464 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 462706 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 462713 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    ] o.a.s.c.CoreContainer Reloading SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1
   [junit4]   2> 462732 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c76aaa2
   [junit4]   2> 462733 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.update.HardAutoCommitTest_389D3A4D00E5B7D1-001/init-core-data-001/]
   [junit4]   2> 462824 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 462825 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 462826 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@26cd1338[collection1] main]
   [junit4]   2> 462827 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 462827 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 462831 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 462831 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570792281474072576
   [junit4]   2> 462833 INFO  (searcherExecutor-1178-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 462834 INFO  (searcherExecutor-1178-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 462834 INFO  (searcherExecutor-1178-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 462834 INFO  (searcherExecutor-1178-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 462838 INFO  (searcherExecutor-1178-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 462838 INFO  (searcherExecutor-1178-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 462839 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=8, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 462839 INFO  (searcherExecutor-1178-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 462840 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 462840 INFO  (searcherExecutor-1178-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 462840 INFO  (searcherExecutor-1178-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 462843 INFO  (searcherExecutor-1178-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@26cd1338[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 462843 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3648aef2[collection1] main]
   [junit4]   2> 462843 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@20141030
   [junit4]   2> 462844 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=538185776
   [junit4]   2> 462853 INFO  (searcherExecutor-1178-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3648aef2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 462858 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[529 (1570792281499238400)]} 0 7
   [junit4]   2> 462859 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:529&qt=standard&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 462862 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[530 (1570792281506578432)]} 0 11
   [junit4]   2> 462863 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:530&qt=standard&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 462863 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {delete=[529 (-1570792281507627008)]} 0 12
   [junit4]   2> 462865 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:529&qt=standard&start=0&rows=20&version=2.2} hits=0 status=0 QTime=1
   [junit4]   2> 462866 ERROR (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.SolrTestCaseJ4 REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]   2> xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]   2> </response>
   [junit4]   2>
   [junit4]   2> request was:q=id:529&qt=standard&start=0&rows=20&version=2.2
   [junit4]   2> 462866 ERROR (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.SolrTestCaseJ4 REQUEST FAILED: q=id:529&qt=standard&start=0&rows=20&version=2.2:java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]   2> xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]   2> </response>
   [junit4]   2>
   [junit4]   2> request was:q=id:529&qt=standard&start=0&rows=20&version=2.2
   [junit4]   2> at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:883)
   [junit4]   2> at org.apache.solr.update.HardAutoCommitTest.testCommitWithin(HardAutoCommitTest.java:100)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 462866 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[389D3A4D00E5B7D1]) [    x:collection1] o.a.s.SolrTestCaseJ4 ###Ending testCommitWithin
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HardAutoCommitTest -Dtests.method=testCommitWithin -Dtests.seed=389D3A4D00E5B7D1 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=es-BO -Dtests.timezone=America/Argentina/ComodRivadavia -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.60s J2 | HardAutoCommitTest.testCommitWithin <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Exception during query
   [junit4]    > at __randomizedtesting.SeedInfo.seed([389D3A4D00E5B7D1:824F553583CB59C4]:0)
   [junit4]    > at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:890)
   [junit4]    > at org.apache.solr.update.HardAutoCommitTest.testCommitWithin(HardAutoCommitTest.java:100)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]    > xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]    > <response>
   [junit4]    > <lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]    > </response>
   [junit4]    > request was:q=id:529&qt=standard&start=0&rows=20&version=2.2
   [junit4]    > at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:883)
   [junit4]    > ... 40 more
   [junit4]   2> 462872 INFO  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    x:collection1] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 462872 INFO  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    x:collection1] o.a.s.c.CoreContainer Shutting down CoreContainer instance=528410199
   [junit4]   2> 462872 INFO  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 462873 INFO  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 462874 INFO  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 462875 INFO  (SUITE-HardAutoCommitTest-seed#[389D3A4D00E5B7D1]-worker) [    x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 462875 INFO  (coreCloseExecutor-1182-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1d1b13de
   [junit4]   2> 462876 INFO  (coreCloseExecutor-1182-thread-1-processing-x:collection1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter close.
   [junit4]   2> 462876 INFO  (coreCloseExecutor-1182-thread-1-processing-x:collection1) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@40428594 commitCommandVersion:0
   [junit4]   2> 462882 INFO  (coreCloseExecutor-1182-thread-1-processing-x:collection1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=488313822
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.update.HardAutoCommitTest_389D3A4D00E5B7D1-001
   [junit4]   2> Jun 21, 2017 5:50:55 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=30540, maxDocsPerChunk=842, blockSize=227), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=30540, blockSize=227)), sim=RandomSimilarity(queryNorm=true): {}, locale=es-BO, timezone=America/Argentina/ComodRivadavia
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=254221688,total=516947968
   [junit4]   2> NOTE: All tests run in this JVM: [StatelessScriptUpdateProcessorFactoryTest, MetricUtilsTest, OpenExchangeRatesOrgProviderTest, FullSolrCloudDistribCmdsTest, TestCloudDeleteByQuery, BooleanFieldTest, TestPolicyCloud, TestCloudManagedSchema, TestConfigSetImmutable, LeaderFailoverAfterPartitionTest, TestBinaryResponseWriter, BasicDistributedZk2Test, CdcrReplicationHandlerTest, SimpleFacetsTest, TestCollationFieldDocValues, TestPHPSerializedResponseWriter, AutoCommitTest, TestCollapseQParserPlugin, TestTrackingShardHandlerFactory, TestPerFieldSimilarityWithDefaultOverride, HardAutoCommitTest]
   [junit4] Completed [116/725 (1!)] on J2 in 1.90s, 1 test, 1 error <<< FAILURES!

[...truncated 1429 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_389D3A4D00E5B7D1-001/init-core-data-001
   [junit4]   2> 3232156 WARN  (SUITE-AutoScalingHandlerTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=3595 numCloses=3595
   [junit4]   2> 3232156 INFO  (SUITE-AutoScalingHandlerTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 3232158 INFO  (SUITE-AutoScalingHandlerTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 3232159 INFO  (SUITE-AutoScalingHandlerTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_389D3A4D00E5B7D1-001/tempDir-001
   [junit4]   2> 3232160 INFO  (SUITE-AutoScalingHandlerTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3232161 INFO  (Thread-10473) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3232161 INFO  (Thread-10473) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3232162 ERROR (Thread-10473) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3232261 INFO  (SUITE-AutoScalingHandlerTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:58287
   [junit4]   2> 3232270 INFO  (jetty-launcher-3519-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3232270 INFO  (jetty-launcher-3519-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3232272 INFO  (jetty-launcher-3519-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4fdbfa6e{/solr,null,AVAILABLE}
   [junit4]   2> 3232272 INFO  (jetty-launcher-3519-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6f860149{/solr,null,AVAILABLE}
   [junit4]   2> 3232276 INFO  (jetty-launcher-3519-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@66a0a632{SSL,[ssl, http/1.1]}{127.0.0.1:58577}
   [junit4]   2> 3232276 INFO  (jetty-launcher-3519-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@38d8e592{SSL,[ssl, http/1.1]}{127.0.0.1:50934}
   [junit4]   2> 3232276 INFO  (jetty-launcher-3519-thread-2) [    ] o.e.j.s.Server Started @3239665ms
   [junit4]   2> 3232276 INFO  (jetty-launcher-3519-thread-1) [    ] o.e.j.s.Server Started @3239665ms
   [junit4]   2> 3232276 INFO  (jetty-launcher-3519-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=58577}
   [junit4]   2> 3232277 INFO  (jetty-launcher-3519-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50934}
   [junit4]   2> 3232277 ERROR (jetty-launcher-3519-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3232277 INFO  (jetty-launcher-3519-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 3232277 INFO  (jetty-launcher-3519-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3232277 INFO  (jetty-launcher-3519-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3232277 INFO  (jetty-launcher-3519-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-21T06:37:07.482Z
   [junit4]   2> 3232277 ERROR (jetty-launcher-3519-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3232277 INFO  (jetty-launcher-3519-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 3232278 INFO  (jetty-launcher-3519-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3232278 INFO  (jetty-launcher-3519-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3232278 INFO  (jetty-launcher-3519-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-21T06:37:07.483Z
   [junit4]   2> 3232280 INFO  (jetty-launcher-3519-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3232280 INFO  (jetty-launcher-3519-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3232291 INFO  (jetty-launcher-3519-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58287/solr
   [junit4]   2> 3232291 INFO  (jetty-launcher-3519-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58287/solr
   [junit4]   2> 3232351 INFO  (jetty-launcher-3519-thread-2) [n:127.0.0.1:58577_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3232352 INFO  (jetty-launcher-3519-thread-1) [n:127.0.0.1:50934_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3232352 INFO  (jetty-launcher-3519-thread-2) [n:127.0.0.1:58577_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:58577_solr
   [junit4]   2> 3232354 INFO  (jetty-launcher-3519-thread-1) [n:127.0.0.1:50934_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50934_solr
   [junit4]   2> 3232354 INFO  (jetty-launcher-3519-thread-2) [n:127.0.0.1:58577_solr    ] o.a.s.c.Overseer Overseer (id=98174699265523718-127.0.0.1:58577_solr-n_0000000000) starting
   [junit4]   2> 3232366 INFO  (zkCallback-3531-thread-1-processing-n:127.0.0.1:58577_solr) [n:127.0.0.1:58577_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3232366 INFO  (zkCallback-3530-thread-1-processing-n:127.0.0.1:50934_solr) [n:127.0.0.1:50934_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3232395 INFO  (jetty-launcher-3519-thread-2) [n:127.0.0.1:58577_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58577_solr
   [junit4]   2> 3232396 INFO  (zkCallback-3530-thread-1-processing-n:127.0.0.1:50934_solr) [n:127.0.0.1:50934_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3232404 INFO  (zkCallback-3531-thread-1-processing-n:127.0.0.1:58577_solr) [n:127.0.0.1:58577_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3232482 INFO  (jetty-launcher-3519-thread-2) [n:127.0.0.1:58577_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58577.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13448615
   [junit4]   2> 3232492 INFO  (jetty-launcher-3519-thread-2) [n:127.0.0.1:58577_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58577.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13448615
   [junit4]   2> 3232492 INFO  (jetty-launcher-3519-thread-2) [n:127.0.0.1:58577_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58577.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13448615
   [junit4]   2> 3232493 INFO  (jetty-launcher-3519-thread-2) [n:127.0.0.1:58577_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_389D3A4D00E5B7D1-001/tempDir-001/node2/.
   [junit4]   2> 3232691 INFO  (jetty-launcher-3519-thread-1) [n:127.0.0.1:50934_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50934.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13448615
   [junit4]   2> 3232699 INFO  (jetty-launcher-3519-thread-1) [n:127.0.0.1:50934_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50934.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13448615
   [junit4]   2> 3232699 INFO  (jetty-launcher-3519-thread-1) [n:127.0.0.1:50934_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50934.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13448615
   [junit4]   2> 3232701 INFO  (jetty-launcher-3519-thread-1) [n:127.0.0.1:50934_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_389D3A4D00E5B7D1-001/tempDir-001/node1/.
   [junit4]   2> 3232738 INFO  (SUITE-AutoScalingHandlerTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3232739 INFO  (SUITE-AutoScalingHandlerTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58287/solr ready
   [junit4]   2> 3232780 INFO  (TEST-AutoScalingHandlerTest.testPolicyAndPreferences-seed#[389D3A4D00E5B7D1]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testPolicyAndPreferences
   [junit4]   2> 3232850 ERROR (qtp101931055-27733) [n:127.0.0.1:50934_solr    ] o.a.s.h.RequestHandlerBase java.lang.RuntimeException: cores is only allowed in 'cluster-policy'
   [junit4]   2> at org.apache.solr.cloud.autoscaling.Policy.lambda$null$1(Policy.java:107)
   [junit4]   2> at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
   [junit4]   2> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
   [junit4]   2> at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
   [junit4]   2> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
   [junit4]   2> at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
   [junit4]   2> at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
   [junit4]   2> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
   [junit4]   2> at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.Policy.lambda$new$2(Policy.java:112)
   [junit4]   2> at java.util.HashMap.forEach(HashMap.java:1288)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.Policy.<init>(Policy.java:102)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.AutoScalingHandler.verifyAutoScalingConf(AutoScalingHandler.java:275)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.AutoScalingHandler.zkSetPolicies(AutoScalingHandler.java:224)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleSetPolicies(AutoScalingHandler.java:205)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleRequestBody(AutoScalingHandler.java:104)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)
   [junit4]   2> at org.apache.solr.api.ApiBag$ReqHandlerToApi.call(ApiBag.java:242)
   [junit4]   2> at org.apache.solr.api.V2HttpCall.handleAdmin(V2HttpCall.java:312)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:728)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:509)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:374)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:318)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 3232850 INFO  (qtp101931055-27733) [n:127.0.0.1:50934_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=500 QTime=30
   [junit4]   2> 3232851 ERROR (qtp101931055-27733) [n:127.0.0.1:50934_solr    ] o.a.s.s.HttpSolrCall null:java.lang.RuntimeException: cores is only allowed in 'cluster-policy'
   [junit4]   2> at org.apache.solr.cloud.autoscaling.Policy.lambda$null$1(Policy.java:107)
   [junit4]   2> at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
   [junit4]   2> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
   [junit4]   2> at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
   [junit4]   2> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
   [junit4]   2> at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
   [junit4]   2> at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
   [junit4]   2> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
   [junit4]   2> at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.Policy.lambda$new$2(Policy.java:112)
   [junit4]   2> at java.util.HashMap.forEach(HashMap.java:1288)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.Policy.<init>(Policy.java:102)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.AutoScalingHandler.verifyAutoScalingConf(AutoScalingHandler.java:275)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.AutoScalingHandler.zkSetPolicies(AutoScalingHandler.java:224)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleSetPolicies(AutoScalingHandler.java:205)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleRequestBody(AutoScalingHandler.java:104)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)
   [junit4]   2> at org.apache.solr.api.ApiBag$ReqHandlerToApi.call(ApiBag.java:242)
   [junit4]   2> at org.apache.solr.api.V2HttpCall.handleAdmin(V2HttpCall.java:312)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:728)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:509)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:374)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:318)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 3232852 INFO  (TEST-AutoScalingHandlerTest.testPolicyAndPreferences-seed#[389D3A4D00E5B7D1]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testPolicyAndPreferences
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=AutoScalingHandlerTest -Dtests.method=testPolicyAndPreferences -Dtests.seed=389D3A4D00E5B7D1 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=ar-BH -Dtests.timezone=Australia/NSW -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.11s J1 | AutoScalingHandlerTest.testPolicyAndPreferences <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Adding a policy with 'cores' attribute should not have succeeded.
   [junit4]    > at __randomizedtesting.SeedInfo.seed([389D3A4D00E5B7D1:9955646479A3B1DF]:0)
   [junit4]    > at org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest.testPolicyAndPreferences(AutoScalingHandlerTest.java:83)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 3232890 INFO  (TEST-AutoScalingHandlerTest.testReadApi-seed#[389D3A4D00E5B7D1]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testReadApi
   [junit4]   2> 3232928 INFO  (qtp1960468712-27748) [n:127.0.0.1:58577_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3232930 INFO  (qtp1960468712-27748) [n:127.0.0.1:58577_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58287/solr ready
   [junit4]   2> 3232945 INFO  (qtp1960468712-27744) [n:127.0.0.1:58577_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3232950 INFO  (qtp101931055-27745) [n:127.0.0.1:50934_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3232954 INFO  (qtp1960468712-27748) [n:127.0.0.1:58577_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=33
   [junit4]   2> 3232958 INFO  (qtp1960468712-27734) [n:127.0.0.1:58577_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3232959 INFO  (qtp1960468712-27734) [n:127.0.0.1:58577_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58287/solr ready
   [junit4]   2> 3232962 INFO  (qtp1960468712-27746) [n:127.0.0.1:58577_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=2
   [junit4]   2> 3232965 INFO  (qtp101931055-27733) [n:127.0.0.1:50934_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3232967 INFO  (qtp1960468712-27734) [n:127.0.0.1:58577_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=11
   [junit4]   2> 3232971 INFO  (qtp1960468712-27736) [n:127.0.0.1:58577_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3232972 INFO  (qtp1960468712-27736) [n:127.0.0.1:58577_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58287/solr ready
   [junit4]   2> 3232973 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3232974 INFO  (qtp101931055-27735) [n:127.0.0.1:50934_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3232975 INFO  (qtp1960468712-27736) [n:127.0.0.1:58577_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=7
   [junit4]   2> 3232976 INFO  (qtp1960468712-27742) [n:127.0.0.1:58577_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 3232979 INFO  (qtp1960468712-27748) [n:127.0.0.1:58577_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3232980 INFO  (qtp1960468712-27748) [n:127.0.0.1:58577_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58287/solr ready
   [junit4]   2> 3232982 INFO  (qtp1960468712-27744) [n:127.0.0.1:58577_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3232983 INFO  (qtp101931055-27741) [n:127.0.0.1:50934_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3232984 INFO  (qtp1960468712-27748) [n:127.0.0.1:58577_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling/diagnostics params={wt=javabin&version=2} status=0 QTime=7
   [junit4]   2> 3232985 INFO  (qtp1960468712-27746) [n:127.0.0.1:58577_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=6&maxShardsPerNode=10&name=readApiTestViolations&nrtReplicas=6&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3232991 INFO  (OverseerThreadFactory-9188-thread-1-processing-n:127.0.0.1:58577_solr) [n:127.0.0.1:58577_solr    ] o.a.s.c.CreateCollectionCmd Create collection readApiTestViolations
   [junit4]   2> 3232991 INFO  (OverseerThreadFactory-9188-thread-1-processing-n:127.0.0.1:58577_solr) [n:127.0.0.1:58577_solr    ] o.a.s.c.CreateCollectionCmd Only one config set found in zk - using it:conf
   [junit4]   2> 3232991 WARN  (OverseerThreadFactory-9188-thread-1-processing-n:127.0.0.1:58577_solr) [n:127.0.0.1:58577_solr    ] o.a.s.c.CreateCollectionCmd Specified number of replicas of 6 on collection readApiTestViolations is higher than the number of Solr instances currently live or live and part of your createNodeSet(2). It's unusual to run two replica of the same slice on the same Solr-instance.
   [junit4]   2> 3233115 INFO  (qtp101931055-27745) [n:127.0.0.1:50934_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=readApiTestViolations_shard1_replica_n6&action=CREATE&numShards=1&collection=readApiTestViolations&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3233115 INFO  (qtp101931055-27745) [n:127.0.0.1:50934_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3233117 INFO  (qtp1960468712-27736) [n:127.0.0.1:58577_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=readApiTestViolations_shard1_replica_n5&action=CREATE&numShards=1&collection=readApiTestViolations&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3233117 INFO  (qtp1960468712-27736) [n:127.0.0.1:58577_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3233119 INFO  (qtp1960468712-27734) [n:127.0.0.1:58577_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=readApiTestViolations_shard1_replica_n3&action=CREATE&numShards=1&collection=readApiTestViolations&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3233123 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=readApiTestViolations_shard1_replica_n1&action=CREATE&numShards=1&collection=readApiTestViolations&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3233125 INFO  (qtp101931055-27733) [n:127.0.0.1:50934_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=readApiTestViolations_shard1_replica_n4&action=CREATE&numShards=1&collection=readApiTestViolations&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3233128 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=readApiTestViolations_shard1_replica_n2&action=CREATE&numShards=1&collection=readApiTestViolations&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3233269 INFO  (zkCallback-3530-thread-1-processing-n:127.0.0.1:50934_solr) [n:127.0.0.1:50934_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/readApiTestViolations/state.json] for collection [readApiTestViolations] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3233269 INFO  (zkCallback-3531-thread-1-processing-n:127.0.0.1:58577_solr) [n:127.0.0.1:58577_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/readApiTestViolations/state.json] for collection [readApiTestViolations] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3234144 INFO  (qtp1960468712-27736) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3234145 INFO  (qtp101931055-27733) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3234151 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3234153 INFO  (qtp101931055-27745) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3234179 INFO  (qtp101931055-27733) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n4] o.a.s.s.IndexSchema [readApiTestViolations_shard1_replica_n4] Schema name=minimal
   [junit4]   2> 3234181 INFO  (qtp1960468712-27734) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3234181 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.s.IndexSchema [readApiTestViolations_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 3234183 INFO  (qtp101931055-27745) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n6] o.a.s.s.IndexSchema [readApiTestViolations_shard1_replica_n6] Schema name=minimal
   [junit4]   2> 3234184 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3234185 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3234185 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'readApiTestViolations_shard1_replica_n2' using configuration from collection readApiTestViolations, trusted=true
   [junit4]   2> 3234185 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50934.solr.core.readApiTestViolations.shard1.replica_n2' (registry 'solr.core.readApiTestViolations.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13448615
   [junit4]   2> 3234186 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3234186 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.c.SolrCore [[readApiTestViolations_shard1_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_389D3A4D00E5B7D1-001/tempDir-001/node1/readApiTestViolations_shard1_replica_n2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_389D3A4D00E5B7D1-001/tempDir-001/node1/./readApiTestViolations_shard1_replica_n2/data/]
   [junit4]   2> 3234186 INFO  (qtp101931055-27745) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3234186 INFO  (qtp101931055-27745) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'readApiTestViolations_shard1_replica_n6' using configuration from collection readApiTestViolations, trusted=true
   [junit4]   2> 3234187 INFO  (qtp101931055-27745) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50934.solr.core.readApiTestViolations.shard1.replica_n6' (registry 'solr.core.readApiTestViolations.shard1.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13448615
   [junit4]   2> 3234187 INFO  (qtp101931055-27745) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3234187 INFO  (qtp101931055-27745) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n6] o.a.s.c.SolrCore [[readApiTestViolations_shard1_replica_n6] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_389D3A4D00E5B7D1-001/tempDir-001/node1/readApiTestViolations_shard1_replica_n6], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_389D3A4D00E5B7D1-001/tempDir-001/node1/./readApiTestViolations_shard1_replica_n6/data/]
   [junit4]   2> 3234191 INFO  (qtp1960468712-27734) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n3] o.a.s.s.IndexSchema [readApiTestViolations_shard1_replica_n3] Schema name=minimal
   [junit4]   2> 3234191 INFO  (qtp1960468712-27736) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n5] o.a.s.s.IndexSchema [readApiTestViolations_shard1_replica_n5] Schema name=minimal
   [junit4]   2> 3234192 INFO  (qtp101931055-27733) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3234192 INFO  (qtp101931055-27733) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'readApiTestViolations_shard1_replica_n4' using configuration from collection readApiTestViolations, trusted=true
   [junit4]   2> 3234192 INFO  (qtp101931055-27733) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50934.solr.core.readApiTestViolations.shard1.replica_n4' (registry 'solr.core.readApiTestViolations.shard1.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13448615
   [junit4]   2> 3234193 INFO  (qtp101931055-27733) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3234193 INFO  (qtp101931055-27733) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n4] o.a.s.c.SolrCore [[readApiTestViolations_shard1_replica_n4] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_389D3A4D00E5B7D1-001/tempDir-001/node1/readApiTestViolations_shard1_replica_n4], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_389D3A4D00E5B7D1-001/tempDir-001/node1/./readApiTestViolations_shard1_replica_n4/data/]
   [junit4]   2> 3234194 INFO  (qtp1960468712-27734) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n3] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3234195 INFO  (qtp1960468712-27734) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n3] o.a.s.c.CoreContainer Creating SolrCore 'readApiTestViolations_shard1_replica_n3' using configuration from collection readApiTestViolations, trusted=true
   [junit4]   2> 3234195 INFO  (qtp1960468712-27734) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58577.solr.core.readApiTestViolations.shard1.replica_n3' (registry 'solr.core.readApiTestViolations.shard1.replica_n3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13448615
   [junit4]   2> 3234195 INFO  (qtp1960468712-27734) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3234195 INFO  (qtp1960468712-27734) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n3] o.a.s.c.SolrCore [[readApiTestViolations_shard1_replica_n3] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_389D3A4D00E5B7D1-001/tempDir-001/node2/readApiTestViolations_shard1_replica_n3], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_389D3A4D00E5B7D1-001/tempDir-001/node2/./readApiTestViolations_shard1_replica_n3/data/]
   [junit4]   2> 3234196 INFO  (qtp1960468712-27736) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n5] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3234196 INFO  (qtp1960468712-27736) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n5] o.a.s.c.CoreContainer Creating SolrCore 'readApiTestViolations_shard1_replica_n5' using configuration from collection readApiTestViolations, trusted=true
   [junit4]   2> 3234197 INFO  (qtp1960468712-27736) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n5] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58577.solr.core.readApiTestViolations.shard1.replica_n5' (registry 'solr.core.readApiTestViolations.shard1.replica_n5') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13448615
   [junit4]   2> 3234197 INFO  (qtp1960468712-27736) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n5] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3234197 INFO  (qtp1960468712-27736) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n5] o.a.s.c.SolrCore [[readApiTestViolations_shard1_replica_n5] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_389D3A4D00E5B7D1-001/tempDir-001/node2/readApiTestViolations_shard1_replica_n5], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_389D3A4D00E5B7D1-001/tempDir-001/node2/./readApiTestViolations_shard1_replica_n5/data/]
   [junit4]   2> 3234198 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n1] o.a.s.s.IndexSchema [readApiTestViolations_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 3234201 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3234201 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'readApiTestViolations_shard1_replica_n1' using configuration from collection readApiTestViolations, trusted=true
   [junit4]   2> 3234203 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58577.solr.core.readApiTestViolations.shard1.replica_n1' (registry 'solr.core.readApiTestViolations.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13448615
   [junit4]   2> 3234203 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3234203 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n1] o.a.s.c.SolrCore [[readApiTestViolations_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_389D3A4D00E5B7D1-001/tempDir-001/node2/readApiTestViolations_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_389D3A4D00E5B7D1-001/tempDir-001/node2/./readApiTestViolations_shard1_replica_n1/data/]
   [junit4]   2> 3234320 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3234320 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3234326 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3234326 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3234328 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@5813008b[readApiTestViolations_shard1_replica_n2] main]
   [junit4]   2> 3234334 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 3234337 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 3234341 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3234342 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570795190534275072
   [junit4]   2> 3234345 INFO  (searcherExecutor-9193-thread-1-processing-n:127.0.0.1:50934_solr x:readApiTestViolations_shard1_replica_n2 s:shard1 c:readApiTestViolations) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.c.SolrCore [readApiTestViolations_shard1_replica_n2] Registered new searcher Searcher@5813008b[readApiTestViolations_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3234353 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3234353 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3234355 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3234355 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3234357 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@18fd235[readApiTestViolations_shard1_replica_n1] main]
   [junit4]   2> 3234360 INFO  (qtp101931055-27739) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=6 found=1 timeoutin=9998ms
   [junit4]   2> 3234361 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 3234361 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 3234362 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3234365 INFO  (qtp1960468712-27736) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n5] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3234365 INFO  (qtp1960468712-27740) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570795190558392320
   [junit4]   2> 3234365 INFO  (qtp1960468712-27736) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n5] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3234366 INFO  (qtp1960468712-27736) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n5] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3234367 INFO  (qtp1960468712-27736) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n5] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3234368 INFO  (searcherExecutor-9198-thread-1-processing-n:127.0.0.1:58577_solr x:readApiTestViolations_shard1_replica_n1 s:shard1 c:readApiTestViolations) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n1] o.a.s.c.SolrCore [readApiTestViolations_shard1_replica_n1] Registered new searcher Searcher@18fd235[readApiTestViolations_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3234369 INFO  (qtp1960468712-27736) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1  x:readApiTestViolations_shard1_replica_n5] o.a.s.s.SolrIndexSearcher Opening [Searcher@3e788238[readApiTestViolations_shard1_replica_n5]

[...truncated too long message...]

it4]   2> 3236495 INFO  (coreCloseExecutor-9223-thread-1) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1 r:core_node5 x:readApiTestViolations_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.readApiTestViolations.shard1.leader, tag=520446129
   [junit4]   2> 3236496 INFO  (coreCloseExecutor-9223-thread-2) [n:127.0.0.1:50934_solr c:readApiTestViolations s:shard1 r:core_node4 x:readApiTestViolations_shard1_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.readApiTestViolations.shard1.leader, tag=1943331541
   [junit4]   2> 3236496 INFO  (coreCloseExecutor-9224-thread-3) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1 r:core_node3 x:readApiTestViolations_shard1_replica_n3] o.a.s.c.SolrCore [readApiTestViolations_shard1_replica_n3]  CLOSING SolrCore org.apache.solr.core.SolrCore@6bb597e8
   [junit4]   2> 3236498 INFO  (coreCloseExecutor-9224-thread-2) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1 r:core_node2 x:readApiTestViolations_shard1_replica_n5] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.readApiTestViolations.shard1.replica_n5, tag=519357162
   [junit4]   2> 3236502 INFO  (coreCloseExecutor-9224-thread-2) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1 r:core_node2 x:readApiTestViolations_shard1_replica_n5] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.readApiTestViolations.shard1.leader, tag=519357162
   [junit4]   2> 3236502 INFO  (coreCloseExecutor-9224-thread-1) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1 r:core_node6 x:readApiTestViolations_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.readApiTestViolations.shard1.replica_n1, tag=1417576309
   [junit4]   2> 3236505 INFO  (coreCloseExecutor-9224-thread-1) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1 r:core_node6 x:readApiTestViolations_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.readApiTestViolations.shard1.leader, tag=1417576309
   [junit4]   2> 3236506 INFO  (coreCloseExecutor-9224-thread-3) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1 r:core_node3 x:readApiTestViolations_shard1_replica_n3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.readApiTestViolations.shard1.replica_n3, tag=1807063016
   [junit4]   2> 3236509 INFO  (coreCloseExecutor-9224-thread-3) [n:127.0.0.1:58577_solr c:readApiTestViolations s:shard1 r:core_node3 x:readApiTestViolations_shard1_replica_n3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.readApiTestViolations.shard1.leader, tag=1807063016
   [junit4]   2> 3236514 INFO  (jetty-closer-3520-thread-1) [    ] o.a.s.c.Overseer Overseer (id=98174699265523718-127.0.0.1:58577_solr-n_0000000000) closing
   [junit4]   2> 3236515 INFO  (OverseerStateUpdate-98174699265523718-127.0.0.1:58577_solr-n_0000000000) [n:127.0.0.1:58577_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:58577_solr
   [junit4]   2> 3238015 WARN  (zkCallback-3530-thread-1-processing-n:127.0.0.1:50934_solr) [n:127.0.0.1:50934_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 3238016 WARN  (zkCallback-3531-thread-1-processing-n:127.0.0.1:58577_solr) [n:127.0.0.1:58577_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 3238016 INFO  (jetty-closer-3520-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4fdbfa6e{/solr,null,UNAVAILABLE}
   [junit4]   2> 3238016 INFO  (jetty-closer-3520-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6f860149{/solr,null,UNAVAILABLE}
   [junit4]   2> 3238018 ERROR (SUITE-AutoScalingHandlerTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3238018 INFO  (SUITE-AutoScalingHandlerTest-seed#[389D3A4D00E5B7D1]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58287 58287
   [junit4]   2> 3238119 INFO  (Thread-10473) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58287 58287
   [junit4]   2> 3238122 WARN  (Thread-10473) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 8 /solr/aliases.json
   [junit4]   2> 2 /solr/security.json
   [junit4]   2> 2 /solr/configs/conf
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 8 /solr/clusterstate.json
   [junit4]   2> 8 /solr/clusterprops.json
   [junit4]   2> 2 /solr/collections/readApiTestViolations/state.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 8 /solr/live_nodes
   [junit4]   2> 8 /solr/collections
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_389D3A4D00E5B7D1-001
   [junit4]   2> Jun 21, 2017 6:37:13 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=75, maxMBSortInHeap=7.913094376188193, sim=RandomSimilarity(queryNorm=true): {}, locale=ar-BH, timezone=Australia/NSW
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=111741960,total=513802240
   [junit4]   2> NOTE: All tests run in this JVM: [TestXmlQParser, TestBulkSchemaAPI, DocValuesTest, MetricsHandlerTest, TestPartialUpdateDeduplication, FileUtilsTest, DocValuesNotIndexedTest, TestCloudRecovery, AsyncCallRequestStatusResponseTest, HdfsThreadLeakTest, TestFilteredDocIdSet, DistributedQueryComponentOptimizationTest, TestCloudNestedDocsSort, SolrIndexMetricsTest, UpdateRequestProcessorFactoryTest, TestCursorMarkWithoutUniqueKey, TestSolrXml, OverseerCollectionConfigSetProcessorTest, SubstringBytesRefFilterTest, CdcrUpdateLogTest, SegmentsInfoRequestHandlerTest, AtomicUpdateProcessorFactoryTest, TestFuzzyAnalyzedSuggestions, BadComponentTest, DocExpirationUpdateProcessorFactoryTest, TestXIncludeConfig, TestSegmentSorting, TestSweetSpotSimilarityFactory, DeleteStatusTest, SpellCheckCollatorWithCollapseTest, BlockJoinFacetRandomTest, ClusterStateUpdateTest, TestFieldResource, AlternateDirectoryTest, TestEmbeddedSolrServerSchemaAPI, TestUseDocValuesAsStored, TestSolrJ, SliceStateTest, DistanceFunctionTest, TestPerFieldSimilarity, ZkCLITest, TestCharFilters, CustomCollectionTest, SolrInfoBeanTest, SparseHLLTest, TestElisionMultitermQuery, TestLegacyTerms, TestSearcherReuse, TestJsonFacets, QueryParsingTest, TestHashPartitioner, TestSurroundQueryParser, HdfsWriteToMultipleCollectionsTest, ManagedSchemaRoundRobinCloudTest, DistanceUnitsTest, OverseerStatusTest, DistributedFacetPivotLargeTest, TestRestoreCore, SchemaVersionSpecificBehaviorTest, TestFiltering, ReplicationFactorTest, TestInitQParser, TestFileDictionaryLookup, MissingSegmentRecoveryTest, TestTlogReplica, TestUpdate, TestPointFields, ClassificationUpdateProcessorIntegrationTest, CollectionReloadTest, TestRawTransformer, MergeStrategyTest, TestDynamicLoading, SOLR749Test, TestSimpleTrackingShardHandler, SolrCLIZkUtilsTest, TlogReplayBufferedWhileIndexingTest, TestStandardQParsers, SmileWriterTest, TestNamedUpdateProcessors, LoggingHandlerTest, TestStressRecovery, TestManagedResource, TestAddFieldRealTimeGet, TestRTGBase, TestConfigOverlay, SuggesterFSTTest, TestCloudJSONFacetJoinDomain, TestPullReplica, DistribJoinFromCollectionTest, TestSubQueryTransformer, TestSchemaNameResource, PreAnalyzedUpdateProcessorTest, TestNumericRangeQuery64, UtilsToolTest, DistributedFacetPivotSmallTest, TestScoreJoinQPScore, TestDefaultStatsCache, TestDistributedGrouping, DistributedDebugComponentTest, ChaosMonkeySafeLeaderTest, SpellCheckComponentTest, ZkStateReaderTest, BigEndianAscendingWordSerializerTest, TestQueryWrapperFilter, SpatialFilterTest, IndexBasedSpellCheckerTest, TestJavabinTupleStreamParser, DistributedIntervalFacetingTest, RuleEngineTest, V2ApiIntegrationTest, TestPostingsSolrHighlighter, TestFieldCacheSortRandom, DirectSolrConnectionTest, InfixSuggestersTest, TestStressCloudBlindAtomicUpdates, TestStressVersions, TestSystemIdResolver, TestComplexPhraseQParserPlugin, BasicDistributedZkTest, CollectionsAPIDistributedZkTest, LeaderElectionIntegrationTest, ShardRoutingTest, RecoveryZkTest, TestReplicationHandler, CollectionsAPIAsyncDistributedZkTest, ConfigSetsAPITest, CreateCollectionCleanupTest, DeleteReplicaTest, ForceLeaderTest, RemoteQueryErrorTest, ReplaceNodeTest, SSLMigrationTest, SaslZkACLProviderTest, SharedFSAutoReplicaFailoverTest, SharedFSAutoReplicaFailoverUtilsTest, SolrCloudExampleTest, SolrXmlInZkTest, TestAuthenticationFramework, TestCloudPivotFacet, TestCloudPseudoReturnFields, TestCollectionAPI, TestConfigSetsAPI, TestCryptoKeys, TestDeleteCollectionOnDownNodes, TestDownShardTolerantSearch, TestExclusionRuleCollectionAccess, TestLeaderElectionWithEmptyReplica, TestLeaderInitiatedRecoveryThread, TestLocalFSCloudBackupRestore, TestPullReplicaErrorHandling, TestReplicaProperties, TestRequestForwarding, TestSolrCloudWithDelegationTokens, TestSolrCloudWithKerberosAlt, TestTolerantUpdateProcessorRandomCloud, AutoScalingHandlerTest]
   [junit4] Completed [559/725 (2!)] on J1 in 6.77s, 2 tests, 1 failure <<< FAILURES!

[...truncated 43766 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1333 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1333/

5 tests failed.
FAILED:  org.apache.solr.ltr.TestLTROnSolrCloud.testSimpleQuery

Error Message:
Path not found: /responseHeader/status

Stack Trace:
java.lang.RuntimeException: Path not found: /responseHeader/status
        at __randomizedtesting.SeedInfo.seed([1F7659E26DA23C8B:6A7B99FE7DB8FA5E]:0)
        at org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:329)
        at org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:271)
        at org.apache.solr.ltr.TestRerankBase.loadFeature(TestRerankBase.java:286)
        at org.apache.solr.ltr.TestLTROnSolrCloud.loadModelsAndFeatures(TestLTROnSolrCloud.java:184)
        at org.apache.solr.ltr.TestLTROnSolrCloud.setupSolrCluster(TestLTROnSolrCloud.java:138)
        at org.apache.solr.ltr.TestLTROnSolrCloud.setUp(TestLTROnSolrCloud.java:58)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:941)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([BAA083B17AB0FCEC:32F4BC6BD44C9114]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test(ChaosMonkeySafeLeaderWithPullReplicasTest.java:174)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
7 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:     1) Thread[id=5975, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    2) Thread[id=5927, name=StoppableCommitThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)    3) Thread[id=5974, name=StoppableCommitThread-SendThread(127.0.0.1:57019), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    4) Thread[id=6045, name=zkCallback-792-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=6044, name=zkCallback-792-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=5976, name=zkCallback-792-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=6046, name=zkCallback-792-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 7 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:
   1) Thread[id=5975, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   2) Thread[id=5927, name=StoppableCommitThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)
   3) Thread[id=5974, name=StoppableCommitThread-SendThread(127.0.0.1:57019), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   4) Thread[id=6045, name=zkCallback-792-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=6044, name=zkCallback-792-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=5976, name=zkCallback-792-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=6046, name=zkCallback-792-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([BAA083B17AB0FCEC]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=5974, name=StoppableCommitThread-SendThread(127.0.0.1:57019), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    2) Thread[id=6045, name=zkCallback-792-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=6044, name=zkCallback-792-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=5976, name=zkCallback-792-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=6046, name=zkCallback-792-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=5974, name=StoppableCommitThread-SendThread(127.0.0.1:57019), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   2) Thread[id=6045, name=zkCallback-792-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=6044, name=zkCallback-792-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=5976, name=zkCallback-792-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=6046, name=zkCallback-792-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([BAA083B17AB0FCEC]:0)


FAILED:  org.apache.solr.cloud.LeaderElectionIntegrationTest.testSimpleSliceLeaderElection

Error Message:
Expected to see nodes come back collection1 null Live Nodes: [127.0.0.1:47429_solr, 127.0.0.1:54391_solr, 127.0.0.1:50239_solr] Last available state: DocCollection(collection1//collections/collection1/state.json/21)={   "pullReplicas":"0",   "replicationFactor":"1",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node1":{           "core":"collection1_shard1_replica_n1",           "base_url":"https://127.0.0.1:50239/solr",           "node_name":"127.0.0.1:50239_solr",           "state":"active",           "type":"NRT"},         "core_node3":{           "core":"collection1_shard1_replica_n2",           "base_url":"https://127.0.0.1:56460/solr",           "node_name":"127.0.0.1:56460_solr",           "state":"down",           "type":"NRT"},         "core_node4":{           "core":"collection1_shard1_replica_n3",           "base_url":"https://127.0.0.1:59716/solr",           "node_name":"127.0.0.1:59716_solr",           "state":"down",           "type":"NRT"},         "core_node5":{           "core":"collection1_shard1_replica_n4",           "base_url":"https://127.0.0.1:57031/solr",           "node_name":"127.0.0.1:57031_solr",           "state":"down",           "type":"NRT"},         "core_node6":{           "core":"collection1_shard1_replica_n5",           "base_url":"https://127.0.0.1:54391/solr",           "node_name":"127.0.0.1:54391_solr",           "state":"active",           "type":"NRT",           "leader":"true"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node2":{           "core":"collection1_shard2_replica_n1",           "base_url":"https://127.0.0.1:47429/solr",           "node_name":"127.0.0.1:47429_solr",           "state":"active",           "type":"NRT",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Expected to see nodes come back collection1
null
Live Nodes: [127.0.0.1:47429_solr, 127.0.0.1:54391_solr, 127.0.0.1:50239_solr]
Last available state: DocCollection(collection1//collections/collection1/state.json/21)={
  "pullReplicas":"0",
  "replicationFactor":"1",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "core":"collection1_shard1_replica_n1",
          "base_url":"https://127.0.0.1:50239/solr",
          "node_name":"127.0.0.1:50239_solr",
          "state":"active",
          "type":"NRT"},
        "core_node3":{
          "core":"collection1_shard1_replica_n2",
          "base_url":"https://127.0.0.1:56460/solr",
          "node_name":"127.0.0.1:56460_solr",
          "state":"down",
          "type":"NRT"},
        "core_node4":{
          "core":"collection1_shard1_replica_n3",
          "base_url":"https://127.0.0.1:59716/solr",
          "node_name":"127.0.0.1:59716_solr",
          "state":"down",
          "type":"NRT"},
        "core_node5":{
          "core":"collection1_shard1_replica_n4",
          "base_url":"https://127.0.0.1:57031/solr",
          "node_name":"127.0.0.1:57031_solr",
          "state":"down",
          "type":"NRT"},
        "core_node6":{
          "core":"collection1_shard1_replica_n5",
          "base_url":"https://127.0.0.1:54391/solr",
          "node_name":"127.0.0.1:54391_solr",
          "state":"active",
          "type":"NRT",
          "leader":"true"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node2":{
          "core":"collection1_shard2_replica_n1",
          "base_url":"https://127.0.0.1:47429/solr",
          "node_name":"127.0.0.1:47429_solr",
          "state":"active",
          "type":"NRT",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([BAA083B17AB0FCEC:E4944EFF39CF600B]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:269)
        at org.apache.solr.cloud.LeaderElectionIntegrationTest.testSimpleSliceLeaderElection(LeaderElectionIntegrationTest.java:109)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12001 lines...]
   [junit4] Suite: org.apache.solr.cloud.LeaderElectionIntegrationTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionIntegrationTest_BAA083B17AB0FCEC-001/init-core-data-001
   [junit4]   2> 975731 INFO  (SUITE-LeaderElectionIntegrationTest-seed#[BAA083B17AB0FCEC]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 975733 INFO  (SUITE-LeaderElectionIntegrationTest-seed#[BAA083B17AB0FCEC]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 975736 INFO  (TEST-LeaderElectionIntegrationTest.testSimpleSliceLeaderElection-seed#[BAA083B17AB0FCEC]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSimpleSliceLeaderElection
   [junit4]   2> 975737 INFO  (TEST-LeaderElectionIntegrationTest.testSimpleSliceLeaderElection-seed#[BAA083B17AB0FCEC]) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 6 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionIntegrationTest_BAA083B17AB0FCEC-001/tempDir-001
   [junit4]   2> 975737 INFO  (TEST-LeaderElectionIntegrationTest.testSimpleSliceLeaderElection-seed#[BAA083B17AB0FCEC]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 975737 INFO  (Thread-1595) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 975738 INFO  (Thread-1595) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 975751 ERROR (Thread-1595) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 975837 INFO  (TEST-LeaderElectionIntegrationTest.testSimpleSliceLeaderElection-seed#[BAA083B17AB0FCEC]) [    ] o.a.s.c.ZkTestServer start zk server on port:35262
   [junit4]   2> 975849 INFO  (jetty-launcher-676-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 975853 INFO  (jetty-launcher-676-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 975857 INFO  (jetty-launcher-676-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 975859 INFO  (jetty-launcher-676-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 975871 INFO  (jetty-launcher-676-thread-5) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 975872 INFO  (jetty-launcher-676-thread-6) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 975873 INFO  (jetty-launcher-676-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@45568228{/solr,null,AVAILABLE}
   [junit4]   2> 975904 INFO  (jetty-launcher-676-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3f3308c3{/solr,null,AVAILABLE}
   [junit4]   2> 975905 INFO  (jetty-launcher-676-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6cdfd775{SSL,[ssl, http/1.1]}{127.0.0.1:57031}
   [junit4]   2> 975905 INFO  (jetty-launcher-676-thread-1) [    ] o.e.j.s.Server Started @982821ms
   [junit4]   2> 975905 INFO  (jetty-launcher-676-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=57031}
   [junit4]   2> 975905 ERROR (jetty-launcher-676-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 975918 INFO  (jetty-launcher-676-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 975918 INFO  (jetty-launcher-676-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 975918 INFO  (jetty-launcher-676-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 975918 INFO  (jetty-launcher-676-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-22T05:55:47.452Z
   [junit4]   2> 975918 INFO  (jetty-launcher-676-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6197103a{SSL,[ssl, http/1.1]}{127.0.0.1:59716}
   [junit4]   2> 975919 INFO  (jetty-launcher-676-thread-4) [    ] o.e.j.s.Server Started @982835ms
   [junit4]   2> 975919 INFO  (jetty-launcher-676-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59716}
   [junit4]   2> 975920 ERROR (jetty-launcher-676-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 975920 INFO  (jetty-launcher-676-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 975920 INFO  (jetty-launcher-676-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 975920 INFO  (jetty-launcher-676-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 975920 INFO  (jetty-launcher-676-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-22T05:55:47.454Z
   [junit4]   2> 975923 INFO  (jetty-launcher-676-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6208e7d7{/solr,null,AVAILABLE}
   [junit4]   2> 975924 INFO  (jetty-launcher-676-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5d55e7a5{SSL,[ssl, http/1.1]}{127.0.0.1:56460}
   [junit4]   2> 975924 INFO  (jetty-launcher-676-thread-2) [    ] o.e.j.s.Server Started @982840ms
   [junit4]   2> 975924 INFO  (jetty-launcher-676-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=56460}
   [junit4]   2> 975924 ERROR (jetty-launcher-676-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 975924 INFO  (jetty-launcher-676-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 975924 INFO  (jetty-launcher-676-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 975924 INFO  (jetty-launcher-676-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 975924 INFO  (jetty-launcher-676-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-22T05:55:47.458Z
   [junit4]   2> 975936 INFO  (jetty-launcher-676-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7530f55a{/solr,null,AVAILABLE}
   [junit4]   2> 975953 INFO  (jetty-launcher-676-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 975954 INFO  (jetty-launcher-676-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 975961 INFO  (jetty-launcher-676-thread-6) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1f718cfc{/solr,null,AVAILABLE}
   [junit4]   2> 975962 INFO  (jetty-launcher-676-thread-6) [    ] o.e.j.s.AbstractConnector Started ServerConnector@53d8f981{SSL,[ssl, http/1.1]}{127.0.0.1:47429}
   [junit4]   2> 975962 INFO  (jetty-launcher-676-thread-6) [    ] o.e.j.s.Server Started @982878ms
   [junit4]   2> 975962 INFO  (jetty-launcher-676-thread-6) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=47429}
   [junit4]   2> 975962 ERROR (jetty-launcher-676-thread-6) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 975962 INFO  (jetty-launcher-676-thread-6) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 975962 INFO  (jetty-launcher-676-thread-6) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 975962 INFO  (jetty-launcher-676-thread-6) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 975962 INFO  (jetty-launcher-676-thread-6) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-22T05:55:47.496Z
   [junit4]   2> 975968 INFO  (jetty-launcher-676-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@58bcd953{SSL,[ssl, http/1.1]}{127.0.0.1:50239}
   [junit4]   2> 975968 INFO  (jetty-launcher-676-thread-3) [    ] o.e.j.s.Server Started @982884ms
   [junit4]   2> 975968 INFO  (jetty-launcher-676-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50239}
   [junit4]   2> 975968 ERROR (jetty-launcher-676-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 975968 INFO  (jetty-launcher-676-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 975968 INFO  (jetty-launcher-676-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 975968 INFO  (jetty-launcher-676-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 975968 INFO  (jetty-launcher-676-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-22T05:55:47.502Z
   [junit4]   2> 975977 INFO  (jetty-launcher-676-thread-5) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1387274a{/solr,null,AVAILABLE}
   [junit4]   2> 975978 INFO  (jetty-launcher-676-thread-5) [    ] o.e.j.s.AbstractConnector Started ServerConnector@657054d7{SSL,[ssl, http/1.1]}{127.0.0.1:54391}
   [junit4]   2> 975978 INFO  (jetty-launcher-676-thread-5) [    ] o.e.j.s.Server Started @982894ms
   [junit4]   2> 975978 INFO  (jetty-launcher-676-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=54391}
   [junit4]   2> 975978 ERROR (jetty-launcher-676-thread-5) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 975978 INFO  (jetty-launcher-676-thread-5) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 975978 INFO  (jetty-launcher-676-thread-5) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 975978 INFO  (jetty-launcher-676-thread-5) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 975978 INFO  (jetty-launcher-676-thread-5) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-22T05:55:47.512Z
   [junit4]   2> 975988 INFO  (jetty-launcher-676-thread-6) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 975989 INFO  (jetty-launcher-676-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 975992 INFO  (jetty-launcher-676-thread-5) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 975994 INFO  (jetty-launcher-676-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35262/solr
   [junit4]   2> 976002 INFO  (jetty-launcher-676-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35262/solr
   [junit4]   2> 976007 INFO  (jetty-launcher-676-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35262/solr
   [junit4]   2> 976013 INFO  (jetty-launcher-676-thread-6) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35262/solr
   [junit4]   2> 976026 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cce5eefc90008, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 976031 INFO  (jetty-launcher-676-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 976032 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cce5eefc90009, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 976047 INFO  (jetty-launcher-676-thread-5) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35262/solr
   [junit4]   2> 976068 INFO  (jetty-launcher-676-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35262/solr
   [junit4]   2> 976091 INFO  (jetty-launcher-676-thread-2) [n:127.0.0.1:56460_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 976095 INFO  (jetty-launcher-676-thread-6) [n:127.0.0.1:47429_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 976091 INFO  (jetty-launcher-676-thread-4) [n:127.0.0.1:59716_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 976105 INFO  (jetty-launcher-676-thread-2) [n:127.0.0.1:56460_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:56460_solr
   [junit4]   2> 976107 INFO  (jetty-launcher-676-thread-6) [n:127.0.0.1:47429_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47429_solr
   [junit4]   2> 976107 INFO  (jetty-launcher-676-thread-4) [n:127.0.0.1:59716_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59716_solr
   [junit4]   2> 976108 INFO  (jetty-launcher-676-thread-2) [n:127.0.0.1:56460_solr    ] o.a.s.c.Overseer Overseer (id=98180199040483340-127.0.0.1:56460_solr-n_0000000000) starting
   [junit4]   2> 976121 INFO  (zkCallback-698-thread-1-processing-n:127.0.0.1:59716_solr) [n:127.0.0.1:59716_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 976121 INFO  (zkCallback-699-thread-1-processing-n:127.0.0.1:56460_solr) [n:127.0.0.1:56460_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 976141 INFO  (zkCallback-703-thread-1-processing-n:127.0.0.1:47429_solr) [n:127.0.0.1:47429_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 976146 INFO  (jetty-launcher-676-thread-1) [n:127.0.0.1:57031_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 976147 INFO  (jetty-launcher-676-thread-3) [n:127.0.0.1:50239_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 976148 INFO  (jetty-launcher-676-thread-5) [n:127.0.0.1:54391_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 976149 INFO  (jetty-launcher-676-thread-3) [n:127.0.0.1:50239_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 976150 INFO  (jetty-launcher-676-thread-5) [n:127.0.0.1:54391_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 976151 INFO  (jetty-launcher-676-thread-1) [n:127.0.0.1:57031_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 976153 INFO  (jetty-launcher-676-thread-3) [n:127.0.0.1:50239_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50239_solr
   [junit4]   2> 976153 INFO  (jetty-launcher-676-thread-5) [n:127.0.0.1:54391_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54391_solr
   [junit4]   2> 976154 INFO  (zkCallback-698-thread-1-processing-n:127.0.0.1:59716_solr) [n:127.0.0.1:59716_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 976154 INFO  (zkCallback-699-thread-1-processing-n:127.0.0.1:56460_solr) [n:127.0.0.1:56460_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 976154 INFO  (zkCallback-703-thread-1-processing-n:127.0.0.1:47429_solr) [n:127.0.0.1:47429_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 976155 INFO  (zkCallback-699-thread-1-processing-n:127.0.0.1:56460_solr) [n:127.0.0.1:56460_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 976155 INFO  (zkCallback-698-thread-1-processing-n:127.0.0.1:59716_solr) [n:127.0.0.1:59716_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 976155 INFO  (zkCallback-703-thread-1-processing-n:127.0.0.1:47429_solr) [n:127.0.0.1:47429_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 976191 INFO  (zkCallback-709-thread-1-processing-n:127.0.0.1:54391_solr) [n:127.0.0.1:54391_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (4)
   [junit4]   2> 976191 INFO  (zkCallback-711-thread-1-processing-n:127.0.0.1:50239_solr) [n:127.0.0.1:50239_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (4)
   [junit4]   2> 976200 INFO  (jetty-launcher-676-thread-1) [n:127.0.0.1:57031_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57031_solr
   [junit4]   2> 976202 INFO  (zkCallback-700-thread-1-processing-n:127.0.0.1:57031_solr) [n:127.0.0.1:57031_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (4)
   [junit4]   2> 976220 INFO  (jetty-launcher-676-thread-2) [n:127.0.0.1:56460_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56460_solr
   [junit4]   2> 976220 INFO  (zkCallback-699-thread-1-processing-n:127.0.0.1:56460_solr) [n:127.0.0.1:56460_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 976220 INFO  (zkCallback-711-thread-1-processing-n:127.0.0.1:50239_solr) [n:127.0.0.1:50239_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 976220 INFO  (zkCallback-703-thread-1-processing-n:127.0.0.1:47429_solr) [n:127.0.0.1:47429_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 976220 INFO  (zkCallback-709-thread-1-processing-n:127.0.0.1:54391_solr) [n:127.0.0.1:54391_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 976221 INFO  (zkCallback-698-thread-1-processing-n:127.0.0.1:59716_solr) [n:127.0.0.1:59716_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 976221 INFO  (zkCallback-711-thread-1-processing-n:127.0.0.1:50239_solr) [n:127.0.0.1:50239_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 976221 INFO  (zkCallback-699-thread-1-processing-n:127.0.0.1:56460_solr) [n:127.0.0.1:56460_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 976221 INFO  (zkCallback-709-thread-1-processing-n:127.0.0.1:54391_solr) [n:127.0.0.1:54391_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 976222 INFO  (zkCallback-703-thread-1-processing-n:127.0.0.1:47429_solr) [n:127.0.0.1:47429_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 976222 INFO  (zkCallback-698-thread-1-processing-n:127.0.0.1:59716_solr) [n:127.0.0.1:59716_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 976228 INFO  (zkCallback-700-thread-1-processing-n:127.0.0.1:57031_solr) [n:127.0.0.1:57031_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (6)
   [junit4]   2> 976321 INFO  (jetty-launcher-676-thread-6) [n:127.0.0.1:47429_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_47429.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976324 INFO  (jetty-launcher-676-thread-4) [n:127.0.0.1:59716_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59716.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976336 INFO  (jetty-launcher-676-thread-4) [n:127.0.0.1:59716_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59716.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976336 INFO  (jetty-launcher-676-thread-4) [n:127.0.0.1:59716_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59716.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976337 INFO  (jetty-launcher-676-thread-4) [n:127.0.0.1:59716_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionIntegrationTest_BAA083B17AB0FCEC-001/tempDir-001/node4/.
   [junit4]   2> 976377 INFO  (jetty-launcher-676-thread-6) [n:127.0.0.1:47429_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_47429.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976378 INFO  (jetty-launcher-676-thread-6) [n:127.0.0.1:47429_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_47429.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976379 INFO  (jetty-launcher-676-thread-6) [n:127.0.0.1:47429_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionIntegrationTest_BAA083B17AB0FCEC-001/tempDir-001/node6/.
   [junit4]   2> 976394 INFO  (jetty-launcher-676-thread-5) [n:127.0.0.1:54391_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54391.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976412 INFO  (jetty-launcher-676-thread-5) [n:127.0.0.1:54391_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54391.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976427 INFO  (jetty-launcher-676-thread-5) [n:127.0.0.1:54391_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54391.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976428 INFO  (jetty-launcher-676-thread-5) [n:127.0.0.1:54391_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionIntegrationTest_BAA083B17AB0FCEC-001/tempDir-001/node5/.
   [junit4]   2> 976435 INFO  (jetty-launcher-676-thread-3) [n:127.0.0.1:50239_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50239.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976465 INFO  (jetty-launcher-676-thread-3) [n:127.0.0.1:50239_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50239.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976465 INFO  (jetty-launcher-676-thread-3) [n:127.0.0.1:50239_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50239.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976468 INFO  (jetty-launcher-676-thread-3) [n:127.0.0.1:50239_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionIntegrationTest_BAA083B17AB0FCEC-001/tempDir-001/node3/.
   [junit4]   2> 976470 INFO  (jetty-launcher-676-thread-2) [n:127.0.0.1:56460_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56460.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976472 INFO  (jetty-launcher-676-thread-1) [n:127.0.0.1:57031_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57031.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976484 INFO  (jetty-launcher-676-thread-2) [n:127.0.0.1:56460_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56460.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976485 INFO  (jetty-launcher-676-thread-2) [n:127.0.0.1:56460_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56460.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976486 INFO  (jetty-launcher-676-thread-1) [n:127.0.0.1:57031_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57031.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976486 INFO  (jetty-launcher-676-thread-1) [n:127.0.0.1:57031_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57031.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 976491 INFO  (jetty-launcher-676-thread-2) [n:127.0.0.1:56460_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionIntegrationTest_BAA083B17AB0FCEC-001/tempDir-001/node2/.
   [junit4]   2> 976492 INFO  (jetty-launcher-676-thread-1) [n:127.0.0.1:57031_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionIntegrationTest_BAA083B17AB0FCEC-001/tempDir-001/node1/.
   [junit4]   2> 976688 INFO  (TEST-LeaderElectionIntegrationTest.testSimpleSliceLeaderElection-seed#[BAA083B17AB0FCEC]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (6)
   [junit4]   2> 976690 INFO  (TEST-LeaderElectionIntegrationTest.testSimpleSliceLeaderElection-seed#[BAA083B17AB0FCEC]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35262/solr ready
   [junit4]   2> 976745 INFO  (qtp335529671-5393) [n:127.0.0.1:47429_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf&maxShardsPerNode=1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 976749 INFO  (OverseerThreadFactory-2639-thread-1-processing-n:127.0.0.1:56460_solr) [n:127.0.0.1:56460_solr    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 976865 INFO  (OverseerStateUpdate-98180199040483340-127.0.0.1:56460_solr-n_0000000000) [n:127.0.0.1:56460_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"collection1_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:50239/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 976867 INFO  (OverseerStateUpdate-98180199040483340-127.0.0.1:56460_solr-n_0000000000) [n:127.0.0.1:56460_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"collection1_shard2_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:47429/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 977075 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=collection1_shard2_replica_n1&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 977075 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 977105 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node1&collection.configName=conf&newCollection=true&name=collection1_shard1_replica_n1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 977106 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 977217 INFO  (zkCallback-703-thread-1-processing-n:127.0.0.1:47429_solr) [n:127.0.0.1:47429_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 977217 INFO  (zkCallback-711-thread-1-processing-n:127.0.0.1:50239_solr) [n:127.0.0.1:50239_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 977219 INFO  (zkCallback-711-thread-2-processing-n:127.0.0.1:50239_solr) [n:127.0.0.1:50239_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 977224 INFO  (zkCallback-703-thread-2-processing-n:127.0.0.1:47429_solr) [n:127.0.0.1:47429_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 978094 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 978110 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema [collection1_shard2_replica_n1] Schema name=minimal
   [junit4]   2> 978114 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 978114 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 978115 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_47429.solr.core.collection1.shard2.replica_n1' (registry 'solr.core.collection1.shard2.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 978115 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 978115 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [[collection1_shard2_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionIntegrationTest_BAA083B17AB0FCEC-001/tempDir-001/node6/collection1_shard2_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionIntegrationTest_BAA083B17AB0FCEC-001/tempDir-001/node6/./collection1_shard2_replica_n1/data/]
   [junit4]   2> 978120 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 978128 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema [collection1_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 978130 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 978130 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 978131 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50239.solr.core.collection1.shard1.replica_n1' (registry 'solr.core.collection1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 978131 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 978131 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [[collection1_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionIntegrationTest_BAA083B17AB0FCEC-001/tempDir-001/node3/collection1_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionIntegrationTest_BAA083B17AB0FCEC-001/tempDir-001/node3/./collection1_shard1_replica_n1/data/]
   [junit4]   2> 978174 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 978174 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 978175 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 978175 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 978177 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1d97a78d[collection1_shard2_replica_n1] main]
   [junit4]   2> 978180 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 978181 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 978182 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 978182 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 978182 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 978182 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570883187209404416
   [junit4]   2> 978183 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 978183 INFO  (searcherExecutor-2652-thread-1-processing-n:127.0.0.1:47429_solr x:collection1_shard2_replica_n1 s:shard2 c:collection1 r:core_node2) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [collection1_shard2_replica_n1] Registered new searcher Searcher@1d97a78d[collection1_shard2_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 978183 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 978185 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@634aab4d[collection1_shard1_replica_n1] main]
   [junit4]   2> 978186 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 978187 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 978187 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 978188 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570883187215695872
   [junit4]   2> 978188 INFO  (searcherExecutor-2653-thread-1-processing-n:127.0.0.1:50239_solr x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@634aab4d[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 978192 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 978192 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 978192 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 978197 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 978197 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 978197 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 978199 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:47429/solr/collection1_shard2_replica_n1/ shard2
   [junit4]   2> 978202 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:50239/solr/collection1_shard1_replica_n1/ shard1
   [junit4]   2> 978305 INFO  (zkCallback-703-thread-2-processing-n:127.0.0.1:47429_solr) [n:127.0.0.1:47429_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 978305 INFO  (zkCallback-703-thread-1-processing-n:127.0.0.1:47429_solr) [n:127.0.0.1:47429_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 978305 INFO  (zkCallback-711-thread-2-processing-n:127.0.0.1:50239_solr) [n:127.0.0.1:50239_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 978306 INFO  (zkCallback-711-thread-1-processing-n:127.0.0.1:50239_solr) [n:127.0.0.1:50239_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 978350 WARN  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.ZkController Skipping recovery according to sys prop solrcloud.skip.autorecovery
   [junit4]   2> 978353 WARN  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.ZkController Skipping recovery according to sys prop solrcloud.skip.autorecovery
   [junit4]   2> 978359 INFO  (qtp335529671-5380) [n:127.0.0.1:47429_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=collection1_shard2_replica_n1&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1284
   [junit4]   2> 978359 INFO  (qtp203567805-5373) [n:127.0.0.1:50239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node1&collection.configName=conf&newCollection=true&name=collection1_shard1_replica_n1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1253
   [junit4]   2> 978365 INFO  (qtp335529671-5393) [n:127.0.0.1:47429_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 978463 INFO  (zkCallback-711-thread-1-processing-n:127.0.0.1:50239_solr) [n:127.0.0.1:50239_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 978463 INFO  (zkCallback-711-thread-2-processing-n:127.0.0.1:50239_solr) [n:127.0.0.1:50239_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 978463 INFO  (zkCallback-703-thread-2-processing-n:127.0.0.1:47429_solr) [n:127.0.0.1:47429_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 978463 INFO  (zkCallback-703-thread-1-processing-n:127.0.0.1:47429_solr) [n:127.0.0.1:47429_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 978750 INFO  (OverseerCollectionConfigSetProcessor-98180199040483340-127.0.0.1:56460_solr-n_0000000000) [n:127.0.0.1:56460_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 979366 INFO  (qtp335529671-5393) [n:127.0.0.1:47429_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf&maxShardsPerNode=1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2620
   [junit4]   2> 979367 INFO  (qtp335529671-5401) [n:127.0.0.1:47429_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 979369 INFO  (OverseerThreadFactory-2639-thread-2-processing-n:127.0.0.1:56460_solr) [n:127.0.0.1:56460_solr    ] o.a.s.c.AddReplicaCmd addReplica() : {
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"addreplica"}
   [junit4]   2> 979371 INFO  (OverseerThreadFactory-2639-thread-2-processing-n:127.0.0.1:56460_solr) [n:127.0.0.1:56460_solr    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:56460_solr for creating new replica
   [junit4]   2> 979372 INFO  (OverseerStateUpdate-98180199040483340-127.0.0.1:56460_solr-n_0000000000) [n:127.0.0.1:56460_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"addreplica",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"collection1_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:56460/solr",
   [junit4]   2>   "node_name":"127.0.0.1:56460_solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 979474 INFO  (zkCallback-711-thread-2-processing-n:127.0.0.1:50239_solr) [n:127.0.0.1:50239_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 979474 INFO  (zkCallback-703-thread-2-processing-n:127.0.0.1:47429_solr) [n:127.0.0.1:47429_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 979474 INFO  (zkCallback-711-thread-1-processing-n:127.0.0.1:50239_solr) [n:127.0.0.1:50239_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 979474 INFO  (zkCallback-703-thread-1-processing-n:127.0.0.1:47429_solr) [n:127.0.0.1:47429_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 979603 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&name=collection1_shard1_replica_n2&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 979603 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 979616 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 979624 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n2] o.a.s.s.IndexSchema [collection1_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 979627 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 979627 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n2' using configuration from collection collection1, trusted=true
   [junit4]   2> 979627 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56460.solr.core.collection1.shard1.replica_n2' (registry 'solr.core.collection1.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77a7ea62
   [junit4]   2> 979628 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 979628 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n2] o.a.s.c.SolrCore [[collection1_shard1_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionIntegrationTest_BAA083B17AB0FCEC-001/tempDir-001/node2/collection1_shard1_replica_n2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionIntegrationTest_BAA083B17AB0FCEC-001/tempDir-001/node2/./collection1_shard1_replica_n2/data/]
   [junit4]   2> 979685 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 979685 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 979688 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 979688 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 979689 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@21bdf941[collection1_shard1_replica_n2] main]
   [junit4]   2> 979691 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 979692 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 979693 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 979693 INFO  (qtp544983802-5385) [n:127.0.0.1:56460_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570883188793802752
   [junit4]   2> 979695 INFO  (searcherExecutor-2662-thread-1-processing-n:127.0.0.1:56460_solr x:collection1_shard1_replica_n2 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:56460_solr c:collection1 s:shard1 r:core_node3 x:collection1_

[...truncated too long message...]

g=1375330808
   [junit4]   2> 25536 WARN  (zkCallback-45-thread-3-processing-n:127.0.0.1:44042_solr) [n:127.0.0.1:44042_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 25548 INFO  (zkCallback-40-thread-1-processing-n:127.0.0.1:51305_solr) [n:127.0.0.1:51305_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 25548 INFO  (zkCallback-47-thread-2-processing-n:127.0.0.1:32899_solr) [n:127.0.0.1:32899_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 25548 INFO  (zkCallback-43-thread-3-processing-n:127.0.0.1:33511_solr) [n:127.0.0.1:33511_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 25563 INFO  (jetty-closer-6-thread-6) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1e3be348{/solr,null,UNAVAILABLE}
   [junit4]   2> 25563 INFO  (zkCallback-43-thread-3-processing-n:127.0.0.1:33511_solr) [n:127.0.0.1:33511_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 25564 INFO  (zkCallback-43-thread-1-processing-n:127.0.0.1:33511_solr) [n:127.0.0.1:33511_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 25567 INFO  (zkCallback-40-thread-1-processing-n:127.0.0.1:51305_solr) [n:127.0.0.1:51305_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 25568 INFO  (zkCallback-40-thread-2-processing-n:127.0.0.1:51305_solr) [n:127.0.0.1:51305_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 25569 INFO  (coreCloseExecutor-111-thread-1) [n:127.0.0.1:33511_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard2.leader, tag=100435185
   [junit4]   2> 25578 INFO  (zkCallback-43-thread-2-processing-n:127.0.0.1:33511_solr) [n:127.0.0.1:33511_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 25579 INFO  (zkCallback-40-thread-4-processing-n:127.0.0.1:51305_solr) [n:127.0.0.1:51305_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 25583 INFO  (jetty-closer-6-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@20c099f4{/solr,null,UNAVAILABLE}
   [junit4]   2> 25583 INFO  (coreCloseExecutor-110-thread-1) [n:127.0.0.1:51305_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard2.leader, tag=927063262
   [junit4]   2> 25589 INFO  (zkCallback-43-thread-2-processing-n:127.0.0.1:33511_solr) [n:127.0.0.1:33511_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 25589 INFO  (jetty-closer-6-thread-3) [    ] o.a.s.c.Overseer Overseer (id=98180300533006355-127.0.0.1:33511_solr-n_0000000000) closing
   [junit4]   2> 25590 INFO  (OverseerStateUpdate-98180300533006355-127.0.0.1:33511_solr-n_0000000000) [n:127.0.0.1:33511_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:33511_solr
   [junit4]   2> 25606 WARN  (zkCallback-43-thread-2-processing-n:127.0.0.1:33511_solr) [n:127.0.0.1:33511_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 25607 INFO  (jetty-closer-6-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@70f78625{/solr,null,UNAVAILABLE}
   [junit4]   2> 26906 WARN  (zkCallback-42-thread-1-processing-n:127.0.0.1:42771_solr) [n:127.0.0.1:42771_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 26907 INFO  (jetty-closer-6-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@315f1f91{/solr,null,UNAVAILABLE}
   [junit4]   2> 26911 WARN  (zkCallback-44-thread-1-processing-n:127.0.0.1:48940_solr) [n:127.0.0.1:48940_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 26912 INFO  (jetty-closer-6-thread-5) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@14f02f29{/solr,null,UNAVAILABLE}
   [junit4]   2> 27089 WARN  (zkCallback-40-thread-4-processing-n:127.0.0.1:51305_solr) [n:127.0.0.1:51305_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 27089 INFO  (jetty-closer-6-thread-7) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4088d023{/solr,null,UNAVAILABLE}
   [junit4]   2> 27091 ERROR (TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[1F7659E26DA23C8B]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 27093 INFO  (TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[1F7659E26DA23C8B]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46208 46208
   [junit4]   2> 27227 INFO  (Thread-12) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46208 46208
   [junit4]   2> 27228 WARN  (Thread-12) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 10 /solr/aliases.json
   [junit4]   2> 9 /solr/security.json
   [junit4]   2> 4 /solr/configs/conf1
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 33 /solr/collections/collection1/state.json
   [junit4]   2> 10 /solr/clusterprops.json
   [junit4]   2> 10 /solr/clusterstate.json
   [junit4]   2> 3 /solr/overseer_elect/election/98180300533006355-127.0.0.1:33511_solr-n_0000000000
   [junit4]   2> 2 /solr/overseer_elect/election/98180300533006360-127.0.0.1:32899_solr-n_0000000002
   [junit4]   2> 2 /solr/overseer_elect/election/98180300533006358-127.0.0.1:43528_solr-n_0000000003
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 10 /solr/live_nodes
   [junit4]   2> 10 /solr/collections
   [junit4]   2>
   [junit4]   2> 27229 INFO  (TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[1F7659E26DA23C8B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSimpleQuery
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLTROnSolrCloud -Dtests.method=testSimpleQuery -Dtests.seed=1F7659E26DA23C8B -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=fr-CH -Dtests.timezone=Singapore -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   16.0s J0 | TestLTROnSolrCloud.testSimpleQuery <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Path not found: /responseHeader/status
   [junit4]    > at __randomizedtesting.SeedInfo.seed([1F7659E26DA23C8B:6A7B99FE7DB8FA5E]:0)
   [junit4]    > at org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:329)
   [junit4]    > at org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:271)
   [junit4]    > at org.apache.solr.ltr.TestRerankBase.loadFeature(TestRerankBase.java:286)
   [junit4]    > at org.apache.solr.ltr.TestLTROnSolrCloud.loadModelsAndFeatures(TestLTROnSolrCloud.java:184)
   [junit4]    > at org.apache.solr.ltr.TestLTROnSolrCloud.setupSolrCluster(TestLTROnSolrCloud.java:138)
   [junit4]    > at org.apache.solr.ltr.TestLTROnSolrCloud.setUp(TestLTROnSolrCloud.java:58)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/contrib/solr-ltr/test/J0/temp/solr.ltr.TestLTROnSolrCloud_1F7659E26DA23C8B-001
   [junit4]   2> Jun 22, 2017 6:21:51 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {_version_=FST50, popularity=PostingsFormat(name=LuceneVarGapFixedInterval), description=PostingsFormat(name=Memory), id=PostingsFormat(name=LuceneVarGapFixedInterval), text=PostingsFormat(name=Asserting), title=PostingsFormat(name=Memory)}, docValues:{}, maxPointsInLeafNode=1897, maxMBSortInHeap=6.41759853689495, sim=RandomSimilarity(queryNorm=false): {}, locale=fr-CH, timezone=Singapore
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=283060400,total=343408640
   [junit4]   2> NOTE: All tests run in this JVM: [TestMultipleAdditiveTreesModel, TestLTROnSolrCloud]
   [junit4] Completed [20/33 (1!)] on J0 in 16.73s, 1 test, 1 error <<< FAILURES!

[...truncated 36843 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1334 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1334/

7 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
8 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:     1) Thread[id=48255, name=zkCallback-1500-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=48157, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    3) Thread[id=48174, name=StoppableIndexingThread-SendThread(127.0.0.1:55407), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    4) Thread[id=48156, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    5) Thread[id=48176, name=zkCallback-1500-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=48256, name=zkCallback-1500-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=48254, name=zkCallback-1500-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    8) Thread[id=48175, name=StoppableIndexingThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:
   1) Thread[id=48255, name=zkCallback-1500-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=48157, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   3) Thread[id=48174, name=StoppableIndexingThread-SendThread(127.0.0.1:55407), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   4) Thread[id=48156, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   5) Thread[id=48176, name=zkCallback-1500-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=48256, name=zkCallback-1500-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=48254, name=zkCallback-1500-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=48175, name=StoppableIndexingThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
        at __randomizedtesting.SeedInfo.seed([304B34BAA0DA99AA]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=48255, name=zkCallback-1500-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=48157, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.security.AccessController.doPrivileged(Native Method)         at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)         at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$3$2.write(SlaveMain.java:459)         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)         at java.io.PrintStream.flush(PrintStream.java:338)         at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)         at java.io.PrintStream.write(PrintStream.java:482)         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)         at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)         at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)         at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)         at java.io.PrintStream.write(PrintStream.java:527)         at java.io.PrintStream.print(PrintStream.java:669)         at java.io.PrintStream.println(PrintStream.java:823)         at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)         at java.lang.Throwable.printStackTrace(Throwable.java:658)         at java.lang.Throwable.printStackTrace(Throwable.java:643)         at java.lang.Throwable.printStackTrace(Throwable.java:634)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:121)    3) Thread[id=48174, name=StoppableIndexingThread-SendThread(127.0.0.1:55407), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    4) Thread[id=48156, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]         at org.apache.solr.common.util.JavaBinCodec.writeExternString(JavaBinCodec.java:1072)         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:615)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:362)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:231)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:222)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)    5) Thread[id=48176, name=zkCallback-1500-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=48256, name=zkCallback-1500-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=48254, name=zkCallback-1500-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=48255, name=zkCallback-1500-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=48157, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.security.AccessController.doPrivileged(Native Method)
        at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
        at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$3$2.write(SlaveMain.java:459)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        at java.io.PrintStream.flush(PrintStream.java:338)
        at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
        at java.io.PrintStream.write(PrintStream.java:482)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
        at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
        at java.io.PrintStream.write(PrintStream.java:527)
        at java.io.PrintStream.print(PrintStream.java:669)
        at java.io.PrintStream.println(PrintStream.java:823)
        at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)
        at java.lang.Throwable.printStackTrace(Throwable.java:658)
        at java.lang.Throwable.printStackTrace(Throwable.java:643)
        at java.lang.Throwable.printStackTrace(Throwable.java:634)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:121)
   3) Thread[id=48174, name=StoppableIndexingThread-SendThread(127.0.0.1:55407), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   4) Thread[id=48156, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
        at org.apache.solr.common.util.JavaBinCodec.writeExternString(JavaBinCodec.java:1072)
        at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:615)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
        at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:362)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:231)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:222)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   5) Thread[id=48176, name=zkCallback-1500-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=48256, name=zkCallback-1500-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=48254, name=zkCallback-1500-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([304B34BAA0DA99AA]:0)


FAILED:  org.apache.solr.ltr.TestLTROnSolrCloud.testSimpleQuery

Error Message:
Path not found: /responseHeader/status

Stack Trace:
java.lang.RuntimeException: Path not found: /responseHeader/status
        at __randomizedtesting.SeedInfo.seed([9ADFB093F925742D:EFD2708FE93FB2F8]:0)
        at org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:329)
        at org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:271)
        at org.apache.solr.ltr.TestRerankBase.loadFeature(TestRerankBase.java:286)
        at org.apache.solr.ltr.TestLTROnSolrCloud.loadModelsAndFeatures(TestLTROnSolrCloud.java:184)
        at org.apache.solr.ltr.TestLTROnSolrCloud.setupSolrCluster(TestLTROnSolrCloud.java:138)
        at org.apache.solr.ltr.TestLTROnSolrCloud.setUp(TestLTROnSolrCloud.java:58)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:941)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([304B34BAA0DA99AA:B81F0B600E26F452]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ClusterStateUpdateTest.testCoreRegistration

Error Message:
expected:<http[]://127.0.0.1:52825/s...> but was:<http[s]://127.0.0.1:52825/s...>

Stack Trace:
org.junit.ComparisonFailure: expected:<http[]://127.0.0.1:52825/s...> but was:<http[s]://127.0.0.1:52825/s...>
        at __randomizedtesting.SeedInfo.seed([304B34BAA0DA99AA:8EC05215D9A0979F]:0)
        at org.junit.Assert.assertEquals(Assert.java:125)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at org.apache.solr.cloud.ClusterStateUpdateTest.testCoreRegistration(ClusterStateUpdateTest.java:104)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitWithChaosMonkey

Error Message:
shard2 is not consistent.  Got 26 from http://127.0.0.1:45177/pmc/az/collection1_shard2_replica_t0 (previous client) and got 40 from http://127.0.0.1:42707/pmc/az/collection1_shard2_replica_t1

Stack Trace:
java.lang.AssertionError: shard2 is not consistent.  Got 26 from http://127.0.0.1:45177/pmc/az/collection1_shard2_replica_t0 (previous client) and got 40 from http://127.0.0.1:42707/pmc/az/collection1_shard2_replica_t1
        at __randomizedtesting.SeedInfo.seed([304B34BAA0DA99AA:BB6CE76BE1DC322E]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1319)
        at org.apache.solr.cloud.ShardSplitTest.testSplitWithChaosMonkey(ShardSplitTest.java:474)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=5701, name=testExecutor-1956-thread-7, state=RUNNABLE, group=TGRP-HdfsUnloadDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5701, name=testExecutor-1956-thread-7, state=RUNNABLE, group=TGRP-HdfsUnloadDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at <a href="http://127.0.0.1:36170:">http://127.0.0.1:36170: Cannot create 1 new replicas for collection multiunload given the current number of live nodes and a maxShardsPerNode of 1
        at __randomizedtesting.SeedInfo.seed([304B34BAA0DA99AA]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:233)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:222)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCores$0(BasicDistributedZkTest.java:598)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11626 lines...]
   [junit4] Suite: org.apache.solr.cloud.ShardSplitTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/init-core-data-001
   [junit4]   2> 222127 WARN  (SUITE-ShardSplitTest-seed#[304B34BAA0DA99AA]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 222127 INFO  (SUITE-ShardSplitTest-seed#[304B34BAA0DA99AA]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 222129 INFO  (SUITE-ShardSplitTest-seed#[304B34BAA0DA99AA]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 222129 INFO  (SUITE-ShardSplitTest-seed#[304B34BAA0DA99AA]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /pmc/az
   [junit4]   2> 222133 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 222133 INFO  (Thread-1383) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 222133 INFO  (Thread-1383) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 222136 ERROR (Thread-1383) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 222233 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.ZkTestServer start zk server on port:57279
   [junit4]   2> 222256 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 222262 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 222264 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 222266 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 222267 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 222269 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 222271 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 222272 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 222274 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 222276 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 222277 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 222279 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 222393 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 222394 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1e6eaf9c{/pmc/az,null,AVAILABLE}
   [junit4]   2> 222401 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@683d0fba{HTTP/1.1,[http/1.1]}{127.0.0.1:50070}
   [junit4]   2> 222401 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.e.j.s.Server Started @225252ms
   [junit4]   2> 222401 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/tempDir-001/control/data, hostContext=/pmc/az, hostPort=50070, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/control-001/cores}
   [junit4]   2> 222401 ERROR (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 222401 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 222402 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 222402 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 222402 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-23T10:25:25.102Z
   [junit4]   2> 222406 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 222406 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/control-001/solr.xml
   [junit4]   2> 222412 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 222415 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57279/solr
   [junit4]   2> 222443 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 222444 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:50070_pmc%2Faz
   [junit4]   2> 222445 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.Overseer Overseer (id=98186921563521028-127.0.0.1:50070_pmc%2Faz-n_0000000000) starting
   [junit4]   2> 222454 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50070_pmc%2Faz
   [junit4]   2> 222457 INFO  (zkCallback-229-thread-1-processing-n:127.0.0.1:50070_pmc%2Faz) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 222531 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e
   [junit4]   2> 222537 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e
   [junit4]   2> 222537 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e
   [junit4]   2> 222538 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/control-001/cores
   [junit4]   2> 222559 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 222560 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:57279/solr ready
   [junit4]   2> 222563 INFO  (qtp1503329364-2843) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&createNodeSet=127.0.0.1:50070_pmc%252Faz&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 222567 INFO  (OverseerThreadFactory-989-thread-1-processing-n:127.0.0.1:50070_pmc%2Faz) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 222568 INFO  (OverseerThreadFactory-989-thread-1-processing-n:127.0.0.1:50070_pmc%2Faz) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.CreateCollectionCmd Only one config set found in zk - using it:conf1
   [junit4]   2> 222673 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 222673 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 222778 INFO  (zkCallback-229-thread-1-processing-n:127.0.0.1:50070_pmc%2Faz) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 223691 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 223706 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 223803 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 223834 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 223835 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e
   [junit4]   2> 223835 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 223835 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 223840 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=679371625, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 223872 WARN  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 223965 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 223965 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 223966 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 223966 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 223970 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1738676577, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 223971 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2fe2544c[control_collection_shard1_replica_n1] main]
   [junit4]   2> 223972 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 223972 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 223974 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 223974 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570990746945716224
   [junit4]   2> 223977 INFO  (searcherExecutor-992-thread-1-processing-n:127.0.0.1:50070_pmc%2Faz x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@2fe2544c[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 223982 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 223982 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 223982 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:50070/pmc/az/control_collection_shard1_replica_n1/
   [junit4]   2> 223982 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 223982 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:50070/pmc/az/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 223982 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 223989 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:50070/pmc/az/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 224092 INFO  (zkCallback-229-thread-1-processing-n:127.0.0.1:50070_pmc%2Faz) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 224140 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 224143 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1470
   [junit4]   2> 224148 INFO  (qtp1503329364-2843) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 224246 INFO  (zkCallback-229-thread-1-processing-n:127.0.0.1:50070_pmc%2Faz) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 224569 INFO  (OverseerCollectionConfigSetProcessor-98186921563521028-127.0.0.1:50070_pmc%2Faz-n_0000000000) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 225148 INFO  (qtp1503329364-2843) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&createNodeSet=127.0.0.1:50070_pmc%252Faz&wt=javabin&version=2} status=0 QTime=2585
   [junit4]   2> 225155 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 225156 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:57279/solr ready
   [junit4]   2> 225156 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 225158 INFO  (qtp1503329364-2840) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&tlogReplicas=0&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 225160 INFO  (OverseerThreadFactory-989-thread-2-processing-n:127.0.0.1:50070_pmc%2Faz) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 225161 INFO  (OverseerThreadFactory-989-thread-2-processing-n:127.0.0.1:50070_pmc%2Faz) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.CreateCollectionCmd Only one config set found in zk - using it:conf1
   [junit4]   2> 225161 WARN  (OverseerThreadFactory-989-thread-2-processing-n:127.0.0.1:50070_pmc%2Faz) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 225366 INFO  (qtp1503329364-2840) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 225366 INFO  (qtp1503329364-2840) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&tlogReplicas=0&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=208
   [junit4]   2> 225485 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/shard-1-001 of type NRT
   [junit4]   2> 225485 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 225487 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@70d71f4c{/pmc/az,null,AVAILABLE}
   [junit4]   2> 225487 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@29a1bb8a{HTTP/1.1,[http/1.1]}{127.0.0.1:48501}
   [junit4]   2> 225488 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.e.j.s.Server Started @228339ms
   [junit4]   2> 225488 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/pmc/az, hostPort=48501, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/shard-1-001/cores}
   [junit4]   2> 225488 ERROR (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 225488 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 225488 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 225488 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 225489 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-23T10:25:28.189Z
   [junit4]   2> 225493 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 225493 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/shard-1-001/solr.xml
   [junit4]   2> 225499 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 225502 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57279/solr
   [junit4]   2> 225514 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 225517 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 225520 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48501_pmc%2Faz
   [junit4]   2> 225521 INFO  (zkCallback-229-thread-1-processing-n:127.0.0.1:50070_pmc%2Faz) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 225523 INFO  (zkCallback-236-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 225524 INFO  (zkCallback-241-thread-1-processing-n:127.0.0.1:48501_pmc%2Faz) [n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 225672 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e
   [junit4]   2> 225678 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e
   [junit4]   2> 225678 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e
   [junit4]   2> 225680 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/shard-1-001/cores
   [junit4]   2> 225706 INFO  (qtp1503329364-2841) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:48501_pmc%252Faz&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 225708 INFO  (OverseerThreadFactory-989-thread-3-processing-n:127.0.0.1:50070_pmc%2Faz) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.AddReplicaCmd addReplica() : {
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "node":"127.0.0.1:48501_pmc%2Faz",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"addreplica"}
   [junit4]   2> 225708 INFO  (OverseerThreadFactory-989-thread-3-processing-n:127.0.0.1:50070_pmc%2Faz) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:48501_pmc%2Faz for creating new replica
   [junit4]   2> 225709 INFO  (OverseerCollectionConfigSetProcessor-98186921563521028-127.0.0.1:50070_pmc%2Faz-n_0000000000) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 225711 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 225711 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 226728 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 226742 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema [collection1_shard2_replica_n0] Schema name=test
   [junit4]   2> 226829 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 226849 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n0' using configuration from collection collection1, trusted=true
   [junit4]   2> 226849 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n0' (registry 'solr.core.collection1.shard2.replica_n0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e
   [junit4]   2> 226849 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 226850 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore [[collection1_shard2_replica_n0] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/shard-1-001/cores/collection1_shard2_replica_n0], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/shard-1-001/cores/collection1_shard2_replica_n0/data/]
   [junit4]   2> 226854 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=679371625, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 226866 WARN  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 226920 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 226920 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 226921 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 226921 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 226923 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1738676577, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 226924 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.SolrIndexSearcher Opening [Searcher@36f99ff0[collection1_shard2_replica_n0] main]
   [junit4]   2> 226926 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 226926 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 226927 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 226929 INFO  (searcherExecutor-1003-thread-1-processing-n:127.0.0.1:48501_pmc%2Faz x:collection1_shard2_replica_n0 s:shard2 c:collection1) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore [collection1_shard2_replica_n0] Registered new searcher Searcher@36f99ff0[collection1_shard2_replica_n0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 226929 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570990750044258304
   [junit4]   2> 226936 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 226936 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 226936 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:48501/pmc/az/collection1_shard2_replica_n0/
   [junit4]   2> 226936 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 226936 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy http://127.0.0.1:48501/pmc/az/collection1_shard2_replica_n0/ has no replicas
   [junit4]   2> 226936 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 226941 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:48501/pmc/az/collection1_shard2_replica_n0/ shard2
   [junit4]   2> 227092 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 227095 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1384
   [junit4]   2> 227098 INFO  (qtp1503329364-2841) [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:48501_pmc%252Faz&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1392
   [junit4]   2> 227216 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/shard-2-001 of type NRT
   [junit4]   2> 227217 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 227218 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1696418d{/pmc/az,null,AVAILABLE}
   [junit4]   2> 227219 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6b1eb9aa{HTTP/1.1,[http/1.1]}{127.0.0.1:42883}
   [junit4]   2> 227219 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.e.j.s.Server Started @230070ms
   [junit4]   2> 227219 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/pmc/az, hostPort=42883, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/shard-2-001/cores}
   [junit4]   2> 227222 ERROR (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 227223 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 227223 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 227223 INFO  (TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [    ] o.a.s.s.SolrDispatchFilter \

[...truncated too long message...]

n1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 28581 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cd4ca75cd0024, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 28582 INFO  (zkCallback-78-thread-2-processing-n:127.0.0.1:52790_solr) [n:127.0.0.1:52790_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 28582 INFO  (zkCallback-75-thread-1-processing-n:127.0.0.1:52533_solr) [n:127.0.0.1:52533_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 28594 INFO  (coreCloseExecutor-225-thread-1) [n:127.0.0.1:52533_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=248948481
   [junit4]   2> 28595 INFO  (zkCallback-75-thread-2-processing-n:127.0.0.1:52533_solr) [n:127.0.0.1:52533_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:52533_solr
   [junit4]   2> 28595 INFO  (zkCallback-75-thread-1-processing-n:127.0.0.1:52533_solr) [n:127.0.0.1:52533_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 29928 WARN  (zkCallback-73-thread-2-processing-n:127.0.0.1:48414_solr) [n:127.0.0.1:48414_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 29928 INFO  (jetty-closer-20-thread-11) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2fb48412{/solr,null,UNAVAILABLE}
   [junit4]   2> 29930 WARN  (zkCallback-76-thread-2-processing-n:127.0.0.1:34517_solr) [n:127.0.0.1:34517_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 29931 WARN  (zkCallback-68-thread-2-processing-n:127.0.0.1:60829_solr) [n:127.0.0.1:60829_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 29931 INFO  (jetty-closer-20-thread-8) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@342d4886{/solr,null,UNAVAILABLE}
   [junit4]   2> 29931 INFO  (jetty-closer-20-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5abe9b4f{/solr,null,UNAVAILABLE}
   [junit4]   2> 29931 WARN  (zkCallback-71-thread-2-processing-n:127.0.0.1:36159_solr) [n:127.0.0.1:36159_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 29932 INFO  (jetty-closer-20-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1808bbbc{/solr,null,UNAVAILABLE}
   [junit4]   2> 29990 WARN  (zkCallback-69-thread-1-processing-n:127.0.0.1:60858_solr) [n:127.0.0.1:60858_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 29990 WARN  (zkCallback-80-thread-2-processing-n:127.0.0.1:39013_solr) [n:127.0.0.1:39013_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 29991 INFO  (jetty-closer-20-thread-9) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@23191a50{/solr,null,UNAVAILABLE}
   [junit4]   2> 29991 INFO  (jetty-closer-20-thread-7) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7d2232aa{/solr,null,UNAVAILABLE}
   [junit4]   2> 30080 WARN  (zkCallback-81-thread-2-processing-n:127.0.0.1:32772_solr) [n:127.0.0.1:32772_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /collections/collection1/state.json]
   [junit4]   2> 30080 WARN  (zkCallback-81-thread-1-processing-n:127.0.0.1:32772_solr) [n:127.0.0.1:32772_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /collections/collection1/state.json]
   [junit4]   2> 30080 INFO  (jetty-closer-20-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2d605775{/solr,null,UNAVAILABLE}
   [junit4]   2> 30093 WARN  (zkCallback-78-thread-2-processing-n:127.0.0.1:52790_solr) [n:127.0.0.1:52790_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 30094 INFO  (jetty-closer-20-thread-5) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3f1125f5{/solr,null,UNAVAILABLE}
   [junit4]   2> 30097 WARN  (zkCallback-75-thread-1-processing-n:127.0.0.1:52533_solr) [n:127.0.0.1:52533_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 30098 INFO  (jetty-closer-20-thread-10) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1bb7cdbe{/solr,null,UNAVAILABLE}
   [junit4]   2> 30099 ERROR (TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[9ADFB093F925742D]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 30101 INFO  (TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[9ADFB093F925742D]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:45671 45671
   [junit4]   2> 30189 INFO  (Thread-46) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:45671 45671
   [junit4]   2> 30190 WARN  (Thread-46) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 14 /solr/aliases.json
   [junit4]   2> 14 /solr/clusterprops.json
   [junit4]   2> 13 /solr/security.json
   [junit4]   2> 3 /solr/configs/conf1
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 16 /solr/collections/collection1/state.json
   [junit4]   2> 14 /solr/clusterstate.json
   [junit4]   2> 3 /solr/overseer_elect/election/98187257920159777-127.0.0.1:52790_solr-n_0000000000
   [junit4]   2> 2 /solr/overseer_elect/election/98187257920159779-127.0.0.1:39013_solr-n_0000000004
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 14 /solr/live_nodes
   [junit4]   2> 14 /solr/collections
   [junit4]   2>
   [junit4]   2> 30191 INFO  (TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[9ADFB093F925742D]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSimpleQuery
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLTROnSolrCloud -Dtests.method=testSimpleQuery -Dtests.seed=9ADFB093F925742D -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=es-PA -Dtests.timezone=Pacific/Kwajalein -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   10.3s J0 | TestLTROnSolrCloud.testSimpleQuery <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Path not found: /responseHeader/status
   [junit4]    > at __randomizedtesting.SeedInfo.seed([9ADFB093F925742D:EFD2708FE93FB2F8]:0)
   [junit4]    > at org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:329)
   [junit4]    > at org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:271)
   [junit4]    > at org.apache.solr.ltr.TestRerankBase.loadFeature(TestRerankBase.java:286)
   [junit4]    > at org.apache.solr.ltr.TestLTROnSolrCloud.loadModelsAndFeatures(TestLTROnSolrCloud.java:184)
   [junit4]    > at org.apache.solr.ltr.TestLTROnSolrCloud.setupSolrCluster(TestLTROnSolrCloud.java:138)
   [junit4]    > at org.apache.solr.ltr.TestLTROnSolrCloud.setUp(TestLTROnSolrCloud.java:58)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/contrib/solr-ltr/test/J0/temp/solr.ltr.TestLTROnSolrCloud_9ADFB093F925742D-001
   [junit4]   2> Jun 23, 2017 11:51:07 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {_version_=FST50, popularity=FSTOrd50, description=BlockTreeOrds(blocksize=128), id=FSTOrd50, text=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), title=BlockTreeOrds(blocksize=128)}, docValues:{}, maxPointsInLeafNode=273, maxMBSortInHeap=7.989211293074257, sim=RandomSimilarity(queryNorm=true): {}, locale=es-PA, timezone=Pacific/Kwajalein
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=165330216,total=332922880
   [junit4]   2> NOTE: All tests run in this JVM: [TestLTRReRankingPipeline, TestManagedFeatureStore, TestUserTermScorereQDF, TestFieldLengthFeature, TestLTRWithSort, TestSelectiveWeightCreation, TestLTROnSolrCloud]
   [junit4] Completed [26/33 (1!)] on J0 in 11.12s, 1 test, 1 error <<< FAILURES!

[...truncated 36821 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1335 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1335/

5 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationHandlerTest:     1) Thread[id=4258, name=cdcr-update-log-synchronizer-1128-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=4824, name=cdcr-update-log-synchronizer-1270-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=4503, name=cdcr-update-log-synchronizer-1192-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=5395, name=cdcr-update-log-synchronizer-1335-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=5642, name=cdcr-update-log-synchronizer-1399-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationHandlerTest:
   1) Thread[id=4258, name=cdcr-update-log-synchronizer-1128-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=4824, name=cdcr-update-log-synchronizer-1270-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=4503, name=cdcr-update-log-synchronizer-1192-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=5395, name=cdcr-update-log-synchronizer-1335-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=5642, name=cdcr-update-log-synchronizer-1399-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([B0D1D400E98C113B]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=4258, name=cdcr-update-log-synchronizer-1128-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=4824, name=cdcr-update-log-synchronizer-1270-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=4503, name=cdcr-update-log-synchronizer-1192-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=5395, name=cdcr-update-log-synchronizer-1335-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=5642, name=cdcr-update-log-synchronizer-1399-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=4258, name=cdcr-update-log-synchronizer-1128-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=4824, name=cdcr-update-log-synchronizer-1270-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=4503, name=cdcr-update-log-synchronizer-1192-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=5395, name=cdcr-update-log-synchronizer-1335-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=5642, name=cdcr-update-log-synchronizer-1399-thread-1, state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([B0D1D400E98C113B]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
6 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest:     1) Thread[id=22741, name=cdcr-update-log-synchronizer-6330-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=23035, name=cdcr-update-log-synchronizer-6428-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=23023, name=cdcr-update-log-synchronizer-6422-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=23350, name=cdcr-update-log-synchronizer-6530-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=23362, name=cdcr-update-log-synchronizer-6536-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=22729, name=cdcr-update-log-synchronizer-6324-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 6 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest:
   1) Thread[id=22741, name=cdcr-update-log-synchronizer-6330-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=23035, name=cdcr-update-log-synchronizer-6428-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=23023, name=cdcr-update-log-synchronizer-6422-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=23350, name=cdcr-update-log-synchronizer-6530-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=23362, name=cdcr-update-log-synchronizer-6536-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=22729, name=cdcr-update-log-synchronizer-6324-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([B0D1D400E98C113B]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=22741, name=cdcr-update-log-synchronizer-6330-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=23035, name=cdcr-update-log-synchronizer-6428-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=23023, name=cdcr-update-log-synchronizer-6422-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=23350, name=cdcr-update-log-synchronizer-6530-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=23362, name=cdcr-update-log-synchronizer-6536-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=22729, name=cdcr-update-log-synchronizer-6324-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=22741, name=cdcr-update-log-synchronizer-6330-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=23035, name=cdcr-update-log-synchronizer-6428-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=23023, name=cdcr-update-log-synchronizer-6422-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=23350, name=cdcr-update-log-synchronizer-6530-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=23362, name=cdcr-update-log-synchronizer-6536-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=22729, name=cdcr-update-log-synchronizer-6324-thread-1, state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([B0D1D400E98C113B]:0)


FAILED:  org.apache.solr.ltr.TestLTROnSolrCloud.testSimpleQuery

Error Message:
Path not found: /responseHeader/status

Stack Trace:
java.lang.RuntimeException: Path not found: /responseHeader/status
        at __randomizedtesting.SeedInfo.seed([C7CE671D17FFDB46:B2C3A70107E51D93]:0)
        at org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:329)
        at org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:271)
        at org.apache.solr.ltr.TestRerankBase.loadFeature(TestRerankBase.java:286)
        at org.apache.solr.ltr.TestLTROnSolrCloud.loadModelsAndFeatures(TestLTROnSolrCloud.java:184)
        at org.apache.solr.ltr.TestLTROnSolrCloud.setupSolrCluster(TestLTROnSolrCloud.java:138)
        at org.apache.solr.ltr.TestLTROnSolrCloud.setUp(TestLTROnSolrCloud.java:58)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:941)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11721 lines...]
   [junit4] Suite: org.apache.solr.cloud.CdcrReplicationHandlerTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/init-core-data-001
   [junit4]   2> 385259 WARN  (SUITE-CdcrReplicationHandlerTest-seed#[B0D1D400E98C113B]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 385259 INFO  (SUITE-CdcrReplicationHandlerTest-seed#[B0D1D400E98C113B]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 385261 INFO  (SUITE-CdcrReplicationHandlerTest-seed#[B0D1D400E98C113B]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 385262 INFO  (SUITE-CdcrReplicationHandlerTest-seed#[B0D1D400E98C113B]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /io/hw
   [junit4]   2> 385267 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 385269 INFO  (Thread-1019) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 385269 INFO  (Thread-1019) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 385287 ERROR (Thread-1019) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 385369 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.ZkTestServer start zk server on port:51040
   [junit4]   2> 385387 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-cdcr.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 385390 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 385391 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 385393 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 385395 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 385396 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 385397 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 385399 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 385401 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 385402 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 385404 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 385420 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cd8c1bed50002, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 385420 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testFullReplication
   [junit4]   2> 385645 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 385647 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@344b5dec{/io/hw,null,AVAILABLE}
   [junit4]   2> 385652 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3624c80e{SSL,[ssl, http/1.1]}{127.0.0.1:40256}
   [junit4]   2> 385652 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.e.j.s.Server Started @389523ms
   [junit4]   2> 385652 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {shards=shard1, hostContext=/io/hw, hostPort=40256, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-001/cores}
   [junit4]   2> 385652 ERROR (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 385653 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 385653 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 385653 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 385653 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-24T06:19:55.338Z
   [junit4]   2> 385661 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 385661 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-001/solr.xml
   [junit4]   2> 385671 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 385675 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51040/solr
   [junit4]   2> 385721 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 385725 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:40256_io%2Fhw
   [junit4]   2> 385727 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.Overseer Overseer (id=98191618537226245-127.0.0.1:40256_io%2Fhw-n_0000000000) starting
   [junit4]   2> 385738 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40256_io%2Fhw
   [junit4]   2> 385761 INFO  (zkCallback-1215-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 385823 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab
   [junit4]   2> 385829 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab
   [junit4]   2> 385829 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab
   [junit4]   2> 385830 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-001/cores
   [junit4]   2> 385976 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 385988 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4b68787d{/io/hw,null,AVAILABLE}
   [junit4]   2> 385990 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5129940d{SSL,[ssl, http/1.1]}{127.0.0.1:39054}
   [junit4]   2> 385990 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.e.j.s.Server Started @389860ms
   [junit4]   2> 385990 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {shards=shard2, hostContext=/io/hw, hostPort=39054, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-002/cores}
   [junit4]   2> 385990 ERROR (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 385990 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 385990 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 385990 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 385991 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-24T06:19:55.675Z
   [junit4]   2> 385994 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 385994 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-002/solr.xml
   [junit4]   2> 386000 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 386006 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51040/solr
   [junit4]   2> 386017 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [n:127.0.0.1:39054_io%2Fhw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 386020 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [n:127.0.0.1:39054_io%2Fhw    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 386022 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [n:127.0.0.1:39054_io%2Fhw    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39054_io%2Fhw
   [junit4]   2> 386024 INFO  (zkCallback-1215-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 386030 INFO  (zkCallback-1221-thread-1-processing-n:127.0.0.1:39054_io%2Fhw) [n:127.0.0.1:39054_io%2Fhw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 386139 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [n:127.0.0.1:39054_io%2Fhw    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab
   [junit4]   2> 386145 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [n:127.0.0.1:39054_io%2Fhw    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab
   [junit4]   2> 386146 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [n:127.0.0.1:39054_io%2Fhw    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab
   [junit4]   2> 386147 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [n:127.0.0.1:39054_io%2Fhw    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-002/cores
   [junit4]   2> 386178 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 386179 INFO  (TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:51040/solr ready
   [junit4]   2> 386213 INFO  (qtp454697609-4200) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=1&name=tmp_collection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&createNodeSet=&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 386216 INFO  (OverseerThreadFactory-1108-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.CreateCollectionCmd Create collection tmp_collection
   [junit4]   2> 386216 INFO  (OverseerThreadFactory-1108-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.CreateCollectionCmd Only one config set found in zk - using it:conf1
   [junit4]   2> 386217 WARN  (OverseerThreadFactory-1108-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (tmp_collection) without cores.
   [junit4]   2> 386326 INFO  (qtp454697609-4200) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 386328 INFO  (qtp454697609-4200) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=1&name=tmp_collection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&createNodeSet=&wt=javabin&version=2} status=0 QTime=114
   [junit4]   2> 386330 INFO  (qtp454697609-4201) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:40256_io%252Fhw&action=ADDREPLICA&collection=tmp_collection&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 386337 INFO  (OverseerThreadFactory-1108-thread-2-processing-n:127.0.0.1:40256_io%2Fhw) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.AddReplicaCmd addReplica() : {
   [junit4]   2>   "collection":"tmp_collection",
   [junit4]   2>   "node":"127.0.0.1:40256_io%2Fhw",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"addreplica"}
   [junit4]   2> 386337 INFO  (OverseerCollectionConfigSetProcessor-98191618537226245-127.0.0.1:40256_io%2Fhw-n_0000000000) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 386341 INFO  (OverseerThreadFactory-1108-thread-2-processing-n:127.0.0.1:40256_io%2Fhw) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:40256_io%2Fhw for creating new replica
   [junit4]   2> 386352 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=tmp_collection_shard1_replica_n0&action=CREATE&collection=tmp_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 386352 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 386462 INFO  (zkCallback-1215-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tmp_collection/state.json] for collection [tmp_collection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 387372 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 387386 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.s.IndexSchema [tmp_collection_shard1_replica_n0] Schema name=test
   [junit4]   2> 387494 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 387515 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.c.CoreContainer Creating SolrCore 'tmp_collection_shard1_replica_n0' using configuration from collection tmp_collection, trusted=true
   [junit4]   2> 387516 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.tmp_collection.shard1.replica_n0' (registry 'solr.core.tmp_collection.shard1.replica_n0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab
   [junit4]   2> 387523 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 387523 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.c.SolrCore [[tmp_collection_shard1_replica_n0] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-001/cores/tmp_collection_shard1_replica_n0], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-001/cores/tmp_collection_shard1_replica_n0/data/]
   [junit4]   2> 387527 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@11cb7e9a
   [junit4]   2> 387637 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
   [junit4]   2> 387638 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 387639 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 387639 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 387641 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=77.1005859375, floorSegmentMB=2.068359375, forceMergeDeletesPctAllowed=6.89125328250852, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8958366005586522
   [junit4]   2> 387642 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.s.SolrIndexSearcher Opening [Searcher@26b2563[tmp_collection_shard1_replica_n0] main]
   [junit4]   2> 387643 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 387644 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 387645 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 387651 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.h.CdcrBufferStateManager Created znode /collections/tmp_collection/cdcr/state/buffer
   [junit4]   2> 387656 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.h.CdcrProcessStateManager Created znode /collections/tmp_collection/cdcr/state/process
   [junit4]   2> 387674 INFO  (searcherExecutor-1117-thread-1-processing-n:127.0.0.1:40256_io%2Fhw x:tmp_collection_shard1_replica_n0 s:shard1 c:tmp_collection) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.c.SolrCore [tmp_collection_shard1_replica_n0] Registered new searcher Searcher@26b2563[tmp_collection_shard1_replica_n0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 387674 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1571065899105910784
   [junit4]   2> 387681 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 387681 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 387681 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:40256/io/hw/tmp_collection_shard1_replica_n0/
   [junit4]   2> 387681 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 387681 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.c.SyncStrategy https://127.0.0.1:40256/io/hw/tmp_collection_shard1_replica_n0/ has no replicas
   [junit4]   2> 387681 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 387685 INFO  (zkCallback-1215-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.h.CdcrLeaderStateManager Received new leader state @ tmp_collection:shard1
   [junit4]   2> 387689 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:40256/io/hw/tmp_collection_shard1_replica_n0/ shard1
   [junit4]   2> 387792 INFO  (zkCallback-1215-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tmp_collection/state.json] for collection [tmp_collection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 387840 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 387843 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=tmp_collection_shard1_replica_n0&action=CREATE&collection=tmp_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1491
   [junit4]   2> 387847 INFO  (qtp454697609-4201) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:40256_io%252Fhw&action=ADDREPLICA&collection=tmp_collection&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1516
   [junit4]   2> 387848 INFO  (qtp454697609-4194) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:39054_io%252Fhw&action=ADDREPLICA&collection=tmp_collection&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 387853 INFO  (OverseerCollectionConfigSetProcessor-98191618537226245-127.0.0.1:40256_io%2Fhw-n_0000000000) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 387854 INFO  (OverseerThreadFactory-1108-thread-3-processing-n:127.0.0.1:40256_io%2Fhw) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.AddReplicaCmd addReplica() : {
   [junit4]   2>   "collection":"tmp_collection",
   [junit4]   2>   "node":"127.0.0.1:39054_io%2Fhw",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"addreplica"}
   [junit4]   2> 387855 INFO  (OverseerThreadFactory-1108-thread-3-processing-n:127.0.0.1:40256_io%2Fhw) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:39054_io%2Fhw for creating new replica
   [junit4]   2> 387908 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=tmp_collection_shard1_replica_n1&action=CREATE&collection=tmp_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 387909 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 388013 INFO  (zkCallback-1215-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tmp_collection/state.json] for collection [tmp_collection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 388013 INFO  (zkCallback-1221-thread-1-processing-n:127.0.0.1:39054_io%2Fhw) [n:127.0.0.1:39054_io%2Fhw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tmp_collection/state.json] for collection [tmp_collection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 388928 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 388939 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.s.IndexSchema [tmp_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 389034 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 389062 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'tmp_collection_shard1_replica_n1' using configuration from collection tmp_collection, trusted=true
   [junit4]   2> 389062 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.tmp_collection.shard1.replica_n1' (registry 'solr.core.tmp_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab
   [junit4]   2> 389062 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 389063 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.c.SolrCore [[tmp_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-002/cores/tmp_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-002/cores/tmp_collection_shard1_replica_n1/data/]
   [junit4]   2> 389067 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@290728fb
   [junit4]   2> 389132 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
   [junit4]   2> 389132 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 389133 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 389133 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 389135 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=77.1005859375, floorSegmentMB=2.068359375, forceMergeDeletesPctAllowed=6.89125328250852, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8958366005586522
   [junit4]   2> 389135 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5b1db555[tmp_collection_shard1_replica_n1] main]
   [junit4]   2> 389137 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 389137 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 389138 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 389146 INFO  (searcherExecutor-1123-thread-1-processing-n:127.0.0.1:39054_io%2Fhw x:tmp_collection_shard1_replica_n1 s:shard1 c:tmp_collection) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.c.SolrCore [tmp_collection_shard1_replica_n1] Registered new searcher Searcher@5b1db555[tmp_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 389146 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1571065900649414656
   [junit4]   2> 389151 INFO  (qtp454697609-4197) [n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1 r:core_node1 x:tmp_collection_shard1_replica_n0] o.a.s.c.S.Request [tmp_collection_shard1_replica_n0]  webapp=/io/hw path=/cdcr params={action=LASTPROCESSEDVERSION&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 389156 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.c.ZkController Core needs to recover:tmp_collection_shard1_replica_n1
   [junit4]   2> 389157 INFO  (updateExecutor-1218-thread-1-processing-n:127.0.0.1:39054_io%2Fhw x:tmp_collection_shard1_replica_n1 s:shard1 c:tmp_collection) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1 r:core_node2 x:tmp_collection_shard1_replica_n1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 389157 INFO  (recoveryExecutor-1219-thread-1-processing-n:127.0.0.1:39054_io%2Fhw x:tmp_collection_shard1_replica_n1 s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1 r:core_node2 x:tmp_collection_shard1_replica_n1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 389158 INFO  (recoveryExecutor-1219-thread-1-processing-n:127.0.0.1:39054_io%2Fhw x:tmp_collection_shard1_replica_n1 s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1 r:core_node2 x:tmp_collection_shard1_replica_n1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 389158 INFO  (recoveryExecutor-1219-thread-1-processing-n:127.0.0.1:39054_io%2Fhw x:tmp_collection_shard1_replica_n1 s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1 r:core_node2 x:tmp_collection_shard1_replica_n1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[tmp_collection_shard1_replica_n1]
   [junit4]   2> 389158 INFO  (recoveryExecutor-1219-thread-1-processing-n:127.0.0.1:39054_io%2Fhw x:tmp_collection_shard1_replica_n1 s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1 r:core_node2 x:tmp_collection_shard1_replica_n1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 389158 INFO  (recoveryExecutor-1219-thread-1-processing-n:127.0.0.1:39054_io%2Fhw x:tmp_collection_shard1_replica_n1 s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1 r:core_node2 x:tmp_collection_shard1_replica_n1] o.a.s.c.RecoveryStrategy Publishing state of core [tmp_collection_shard1_replica_n1] as recovering, leader is [https://127.0.0.1:40256/io/hw/tmp_collection_shard1_replica_n0/] and I am [https://127.0.0.1:39054/io/hw/tmp_collection_shard1_replica_n1/]
   [junit4]   2> 389158 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=tmp_collection_shard1_replica_n1&action=CREATE&collection=tmp_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1250
   [junit4]   2> 389164 INFO  (qtp454697609-4194) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:39054_io%252Fhw&action=ADDREPLICA&collection=tmp_collection&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1316
   [junit4]   2> 389164 INFO  (recoveryExecutor-1219-thread-1-processing-n:127.0.0.1:39054_io%2Fhw x:tmp_collection_shard1_replica_n1 s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1 r:core_node2 x:tmp_collection_shard1_replica_n1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [https://127.0.0.1:40256/io/hw]; [WaitForState: action=PREPRECOVERY&core=tmp_collection_shard1_replica_n0&nodeName=127.0.0.1:39054_io%252Fhw&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 389167 INFO  (qtp454697609-4200) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true, maxTime: 183 s
   [junit4]   2> 389171 INFO  (qtp454697609-4200) [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=tmp_collection, shard=shard1, thisCore=tmp_collection_shard1_replica_n0, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:39054_io%2Fhw, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"tmp_collection_shard1_replica_n1","base_url":"https://127.0.0.1:39054/io/hw","node_name":"127.0.0.1:39054_io%2Fhw","state":"down","type":"NRT"}
   [junit4]   1> / (2)
   [junit4]   1> DATA:
   [junit4]   1>    
   [junit4]   1>  /zookeeper (1)
   [junit4]   1>  DATA:
   [junit4]   1>      
   [junit4]   1>  /solr (10)
   [junit4]   1>   /solr/configs (1)
   [junit4]   1>    /solr/configs/conf1 (11)
   [junit4]   1>     /solr/configs/conf1/currency.xml (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/enumsConfig.xml (0)
   [junit4]   1>     DATA: ...s

[...truncated too long message...]

 instance=1045063570
   [junit4]   2> 27211 INFO  (jetty-closer-24-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1746893512
   [junit4]   2> 27217 INFO  (jetty-closer-24-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1460563256
   [junit4]   2> 27218 INFO  (jetty-closer-24-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 27218 INFO  (jetty-closer-24-thread-3) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 27218 INFO  (jetty-closer-24-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 27224 INFO  (jetty-closer-24-thread-3) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 27226 INFO  (jetty-closer-24-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 27229 INFO  (jetty-closer-24-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 27233 INFO  (jetty-closer-24-thread-3) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 27235 INFO  (jetty-closer-24-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 27236 INFO  (jetty-closer-24-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 27238 INFO  (jetty-closer-24-thread-3) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 27239 INFO  (jetty-closer-24-thread-3) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 27239 INFO  (jetty-closer-24-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 27242 INFO  (jetty-closer-24-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@268c9ce3{/solr,null,UNAVAILABLE}
   [junit4]   2> 27242 INFO  (jetty-closer-24-thread-2) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 27242 INFO  (jetty-closer-24-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 27243 INFO  (zkCallback-38-thread-2-processing-n:127.0.0.1:49345_solr) [n:127.0.0.1:49345_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 27243 INFO  (zkCallback-40-thread-2-processing-n:127.0.0.1:50282_solr) [n:127.0.0.1:50282_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 27245 INFO  (coreCloseExecutor-178-thread-1) [n:127.0.0.1:49345_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1c753d9a
   [junit4]   2> 27245 INFO  (jetty-closer-24-thread-2) [    ] o.a.s.c.Overseer Overseer (id=98191805058580489-127.0.0.1:50282_solr-n_0000000000) closing
   [junit4]   2> 27245 INFO  (OverseerStateUpdate-98191805058580489-127.0.0.1:50282_solr-n_0000000000) [n:127.0.0.1:50282_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:50282_solr
   [junit4]   2> 27254 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cd8ed2c5f0009, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 27254 INFO  (zkCallback-38-thread-1-processing-n:127.0.0.1:49345_solr) [n:127.0.0.1:49345_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 27255 INFO  (zkCallback-38-thread-2-processing-n:127.0.0.1:49345_solr) [n:127.0.0.1:49345_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:49345_solr
   [junit4]   2> 27261 INFO  (coreCloseExecutor-178-thread-1) [n:127.0.0.1:49345_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1.shard1.replica_n1, tag=477445530
   [junit4]   2> 27269 INFO  (coreCloseExecutor-178-thread-1) [n:127.0.0.1:49345_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=477445530
   [junit4]   2> 27281 WARN  (zkCallback-38-thread-2-processing-n:127.0.0.1:49345_solr) [n:127.0.0.1:49345_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 27281 INFO  (jetty-closer-24-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@43541600{/solr,null,UNAVAILABLE}
   [junit4]   2> 28753 WARN  (zkCallback-40-thread-2-processing-n:127.0.0.1:50282_solr) [n:127.0.0.1:50282_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 28754 INFO  (jetty-closer-24-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4213cf1e{/solr,null,UNAVAILABLE}
   [junit4]   2> 28755 ERROR (TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[C7CE671D17FFDB46]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 28757 INFO  (TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[C7CE671D17FFDB46]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43036 43036
   [junit4]   2> 28860 INFO  (Thread-57) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43036 43036
   [junit4]   2> 28863 WARN  (Thread-57) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 5 /solr/aliases.json
   [junit4]   2> 5 /solr/clusterprops.json
   [junit4]   2> 4 /solr/security.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 5 /solr/clusterstate.json
   [junit4]   2> 5 /solr/collections/collection1/state.json
   [junit4]   2> 2 /solr/overseer_elect/election/98191805058580489-127.0.0.1:50282_solr-n_0000000000
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 5 /solr/live_nodes
   [junit4]   2> 5 /solr/collections
   [junit4]   2>
   [junit4]   2> 28863 INFO  (TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[C7CE671D17FFDB46]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSimpleQuery
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLTROnSolrCloud -Dtests.method=testSimpleQuery -Dtests.seed=C7CE671D17FFDB46 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=ar-IQ -Dtests.timezone=America/Asuncion -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   9.01s J0 | TestLTROnSolrCloud.testSimpleQuery <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Path not found: /responseHeader/status
   [junit4]    > at __randomizedtesting.SeedInfo.seed([C7CE671D17FFDB46:B2C3A70107E51D93]:0)
   [junit4]    > at org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:329)
   [junit4]    > at org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:271)
   [junit4]    > at org.apache.solr.ltr.TestRerankBase.loadFeature(TestRerankBase.java:286)
   [junit4]    > at org.apache.solr.ltr.TestLTROnSolrCloud.loadModelsAndFeatures(TestLTROnSolrCloud.java:184)
   [junit4]    > at org.apache.solr.ltr.TestLTROnSolrCloud.setupSolrCluster(TestLTROnSolrCloud.java:138)
   [junit4]    > at org.apache.solr.ltr.TestLTROnSolrCloud.setUp(TestLTROnSolrCloud.java:58)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/contrib/solr-ltr/test/J0/temp/solr.ltr.TestLTROnSolrCloud_C7CE671D17FFDB46-001
   [junit4]   2> Jun 24, 2017 7:07:29 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=13190, maxDocsPerChunk=5, blockSize=4), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=13190, blockSize=4)), sim=RandomSimilarity(queryNorm=false): {}, locale=ar-IQ, timezone=America/Asuncion
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=166008960,total=337117184
   [junit4]   2> NOTE: All tests run in this JVM: [TestRankingFeature, TestExternalValueFeatures, TestUserTermScorereQDF, TestFieldLengthFeature, TestManagedFeatureStore, TestNoMatchSolrFeature, TestLTROnSolrCloud]
   [junit4] Completed [29/33 (1!)] on J0 in 9.31s, 1 test, 1 error <<< FAILURES!

[...truncated 36772 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1336 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1336/

5 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsBasicDistributedZkTest.test

Error Message:
Error from server at <a href="http://127.0.0.1:42980:">http://127.0.0.1:42980: ADDREPLICA failed to create replica

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at <a href="http://127.0.0.1:42980:">http://127.0.0.1:42980: ADDREPLICA failed to create replica
        at __randomizedtesting.SeedInfo.seed([E10E9872A25D54AE:695AA7A80CA13956]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:624)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:239)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:470)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:400)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1102)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:843)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:774)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.BasicDistributedZkTest.testANewCollectionInOneInstanceWithManualShardAssignement(BasicDistributedZkTest.java:808)
        at org.apache.solr.cloud.BasicDistributedZkTest.test(BasicDistributedZkTest.java:374)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsBasicDistributedZkTest

Error Message:
3 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsBasicDistributedZkTest:     1) Thread[id=5968, name=Connection evictor, state=TIMED_WAITING, group=TGRP-HdfsBasicDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=5949, name=Connection evictor, state=TIMED_WAITING, group=TGRP-HdfsBasicDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=5957, name=Connection evictor, state=TIMED_WAITING, group=TGRP-HdfsBasicDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 3 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsBasicDistributedZkTest:
   1) Thread[id=5968, name=Connection evictor, state=TIMED_WAITING, group=TGRP-HdfsBasicDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=5949, name=Connection evictor, state=TIMED_WAITING, group=TGRP-HdfsBasicDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=5957, name=Connection evictor, state=TIMED_WAITING, group=TGRP-HdfsBasicDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([E10E9872A25D54AE]:0)


FAILED:  org.apache.solr.update.AutoCommitTest.testMaxDocs

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
        at __randomizedtesting.SeedInfo.seed([E10E9872A25D54AE:588F4EAD8EB75024]:0)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:872)
        at org.apache.solr.update.AutoCommitTest.testMaxDocs(AutoCommitTest.java:225)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
        xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

        request was:q=id:14&qt=standard&start=0&rows=20&version=2.2
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:865)
        ... 40 more


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at __randomizedtesting.SeedInfo.seed([E10E9872A25D54AE:695AA7A80CA13956]:0)
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649)
        at java.lang.StringBuilder.append(StringBuilder.java:202)
        at org.apache.http.client.utils.URLEncodedUtils.urlEncode(URLEncodedUtils.java:522)
        at org.apache.http.client.utils.URLEncodedUtils.encodeFormFields(URLEncodedUtils.java:638)
        at org.apache.http.client.utils.URLEncodedUtils.format(URLEncodedUtils.java:390)
        at org.apache.http.client.utils.URLEncodedUtils.format(URLEncodedUtils.java:369)
        at org.apache.http.client.entity.UrlEncodedFormEntity.<init>(UrlEncodedFormEntity.java:77)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:445)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:239)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:470)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:400)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1102)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:843)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:774)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:974)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:990)
        at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:228)
        at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:167)
        at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:668)
        at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)


FAILED:  org.apache.solr.core.TestJmxIntegration.testJmxRegistration

Error Message:
org.apache.lucene.store.AlreadyClosedException: this Directory is closed

Stack Trace:
javax.management.RuntimeMBeanException: org.apache.lucene.store.AlreadyClosedException: this Directory is closed
        at __randomizedtesting.SeedInfo.seed([E10E9872A25D54AE:6FDFFC48CF1C0CCB]:0)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
        at org.apache.solr.core.TestJmxIntegration.testJmxRegistration(TestJmxIntegration.java:121)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.lucene.store.AlreadyClosedException: this Directory is closed
        at org.apache.lucene.store.BaseDirectory.ensureOpen(BaseDirectory.java:51)
        at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:233)
        at org.apache.solr.core.DirectoryFactory.sizeOfDirectory(DirectoryFactory.java:271)
        at org.apache.solr.core.SolrCore.getIndexSize(SolrCore.java:407)
        at org.apache.solr.core.SolrCore.lambda$initializeMetrics$7(SolrCore.java:1137)
        at com.codahale.metrics.JmxReporter$JmxGauge.getValue(JmxReporter.java:188)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
        at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
        at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
        ... 41 more




Build Log:
[...truncated 11428 lines...]
   [junit4] Suite: org.apache.solr.update.AutoCommitTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.update.AutoCommitTest_E10E9872A25D54AE-001/init-core-data-001
   [junit4]   2> 167920 WARN  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 167921 INFO  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 167923 INFO  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 167928 INFO  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 167929 INFO  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 168092 INFO  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 168168 INFO  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 168619 INFO  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 168725 INFO  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29c3cbc0
   [junit4]   2> 168728 INFO  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29c3cbc0
   [junit4]   2> 168728 INFO  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29c3cbc0
   [junit4]   2> 168758 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 168975 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 169128 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 169772 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 169811 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 169812 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29c3cbc0
   [junit4]   2> 169820 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 169820 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.update.AutoCommitTest_E10E9872A25D54AE-001/init-core-data-001/]
   [junit4]   2> 169825 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1392663930, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 170026 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 170026 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 170027 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 170027 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 170027 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=34, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 170028 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@9ff62e4[collection1] main]
   [junit4]   2> 170029 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 170040 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 170385 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 170386 INFO  (searcherExecutor-163-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 170386 INFO  (coreLoadExecutor-162-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1571152192838041600
   [junit4]   2> 170395 INFO  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 170413 INFO  (searcherExecutor-163-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 170413 INFO  (searcherExecutor-163-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 170413 INFO  (searcherExecutor-163-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 170413 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testMaxTime
   [junit4]   2> 170415 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 170415 INFO  (searcherExecutor-163-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 170416 INFO  (searcherExecutor-163-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 170418 INFO  (searcherExecutor-163-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 170418 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@7490ea5d[collection1] realtime]
   [junit4]   2> 170419 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:* (-9223372036854775807)} 0 5
   [junit4]   2> 170422 INFO  (searcherExecutor-163-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 170422 ERROR (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    ] o.a.s.c.CorePropertiesLocator Couldn't load core descriptor from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/core.properties:java.nio.file.NoSuchFileException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/core.properties
   [junit4]   2> 170423 INFO  (searcherExecutor-163-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 170426 INFO  (searcherExecutor-163-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@9ff62e4[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 170427 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 170670 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 171012 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 171223 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 171292 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    ] o.a.s.c.CoreContainer Reloading SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1
   [junit4]   2> 171357 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29c3cbc0
   [junit4]   2> 171357 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.update.AutoCommitTest_E10E9872A25D54AE-001/init-core-data-001/]
   [junit4]   2> 171569 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 171569 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 171571 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@25f4fb1a[collection1] main]
   [junit4]   2> 171572 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 171574 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 171578 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 171578 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1571152194087944192
   [junit4]   2> 171590 INFO  (searcherExecutor-168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 171590 INFO  (searcherExecutor-168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 171590 INFO  (searcherExecutor-168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 171590 INFO  (searcherExecutor-168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 171591 INFO  (searcherExecutor-168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 171591 INFO  (searcherExecutor-168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 171591 INFO  (searcherExecutor-168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 171592 INFO  (searcherExecutor-168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 171593 INFO  (searcherExecutor-168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 171595 INFO  (searcherExecutor-168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@25f4fb1a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 171607 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@1dfa5e50
   [junit4]   2> 171607 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 171608 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2f888b61[collection1] main]
   [junit4]   2> 171608 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3402011b
   [junit4]   2> 171610 INFO  (searcherExecutor-168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2f888b61[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 171613 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=872546587
   [junit4]   2> 171663 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[529 (1571152194156101632)]} 0 22
   [junit4]   2> 171663 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:529&qt=standard&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 173146 INFO  (commitScheduler-171-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 173176 INFO  (commitScheduler-171-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@20d7a240[collection1] main]
   [junit4]   2> 173176 INFO  (commitScheduler-171-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 173178 INFO  (searcherExecutor-168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@20d7a240[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):c1)))}
   [junit4]   2> 173199 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[530 (1571152195786637312)]} 0 1558
   [junit4]   2> 173199 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:529&qt=standard&start=0&rows=20&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 173200 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:530&qt=standard&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 173201 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{delete=[529 (-1571152195789783040)]} 0 0
   [junit4]   2> 173208 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:529&qt=standard&start=0&rows=20&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 174699 INFO  (commitScheduler-171-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 174712 INFO  (commitScheduler-171-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7f825270[collection1] main]
   [junit4]   2> 174715 INFO  (commitScheduler-171-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 174719 INFO  (searcherExecutor-168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7f825270[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(7.0.0):c1)))}
   [junit4]   2> 174921 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[550 (1571152197588090880)]} 0 3280
   [junit4]   2> 174921 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:529&qt=standard&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 174931 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152197603819520)]} 0 3290
   [junit4]   2> 174931 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152197603819521)]} 0 3291
   [junit4]   2> 174934 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152197604868096)]} 0 3293
   [junit4]   2> 174936 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152197606965248)]} 0 3295
   [junit4]   2> 174937 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152197609062400)]} 0 3296
   [junit4]   2> 174937 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152197610110976)]} 0 3296
   [junit4]   2> 174937 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152197610110977)]} 0 3297
   [junit4]   2> 174947 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152197619548160)]} 0 3306
   [junit4]   2> 174947 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152197620596736)]} 0 3306
   [junit4]   2> 174947 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152197620596737)]} 0 3307
   [junit4]   2> 174948 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:500&qt=standard&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 176422 INFO  (commitScheduler-171-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 176433 INFO  (commitScheduler-171-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3a957d86[collection1] main]
   [junit4]   2> 176435 INFO  (searcherExecutor-168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3a957d86[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(7.0.0):c1) Uninverting(_2(7.0.0):c11/9:delGen=1)))}
   [junit4]   2> 176436 INFO  (commitScheduler-171-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 176496 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[531 (1571152199240646656)]} 0 4856
   [junit4]   2> 176497 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:500&qt=standard&start=0&rows=20&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 176497 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.SolrTestCaseJ4 ###Ending testMaxTime
   [junit4]   2> 176511 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.SolrTestCaseJ4 ###Starting testMaxDocs
   [junit4]   2> 176516 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 176520 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2314542[collection1] realtime]
   [junit4]   2> 176520 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:* (-9223372036854775807)} 0 9
   [junit4]   2> 176521 ERROR (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.CorePropertiesLocator Couldn't load core descriptor from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/core.properties:java.nio.file.NoSuchFileException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/core.properties
   [junit4]   2> 176521 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 176891 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 176938 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 177234 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 177261 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.CoreContainer Reloading SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1
   [junit4]   2> 177290 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29c3cbc0
   [junit4]   2> 177290 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.update.AutoCommitTest_E10E9872A25D54AE-001/init-core-data-001/]
   [junit4]   2> 177570 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 177570 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 177571 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@69670b1a[collection1] main]
   [junit4]   2> 177572 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 177573 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 177582 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 177588 INFO  (searcherExecutor-172-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 177589 INFO  (searcherExecutor-172-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 177589 INFO  (searcherExecutor-172-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 177589 INFO  (searcherExecutor-172-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 177589 INFO  (searcherExecutor-172-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 177592 INFO  (searcherExecutor-172-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 177593 INFO  (searcherExecutor-172-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 177593 INFO  (searcherExecutor-172-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 177593 INFO  (searcherExecutor-172-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 177687 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=29, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=28.3984375, floorSegmentMB=0.3037109375, forceMergeDeletesPctAllowed=19.904743074171964, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.47686791862082534
   [junit4]   2> 177688 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 177800 INFO  (searcherExecutor-172-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@69670b1a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 177800 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@556394ad[collection1] main]
   [junit4]   2> 177800 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6711a381
   [junit4]   2> 177801 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1729209217
   [junit4]   2> 177804 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[0 (1571152200615329792)]} 0 0
   [junit4]   2> 177804 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[1 (1571152200616378368)]} 0 1
   [junit4]   2> 177804 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[2 (1571152200616378369)]} 0 1
   [junit4]   2> 177805 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[3 (1571152200616378370)]} 0 1
   [junit4]   2> 177805 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[4 (1571152200617426944)]} 0 2
   [junit4]   2> 177806 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[5 (1571152200617426945)]} 0 2
   [junit4]   2> 177806 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[6 (1571152200618475520)]} 0 3
   [junit4]   2> 177806 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[7 (1571152200618475521)]} 0 3
   [junit4]   2> 177807 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[8 (1571152200618475522)]} 0 3
   [junit4]   2> 177807 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[9 (1571152200619524096)]} 0 4
   [junit4]   2> 177807 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[10 (1571152200619524097)]} 0 4
   [junit4]   2> 177808 INFO  (searcherExecutor-172-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@556394ad[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 177810 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[11 (1571152200622669824)]} 0 7
   [junit4]   2> 177811 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[12 (1571152200622669825)]} 0 8
   [junit4]   2> 177811 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[13 (1571152200623718400)]} 0 8
   [junit4]   2> 177811 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&qt=standard&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 177825 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[14 (1571152200624766976)]} 0 22
   [junit4]   2> 177825 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[15 (1571152200638398464)]} 0 22
   [junit4]   2> 177826 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:14&qt=standard&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 177826 ERROR (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.SolrTestCaseJ4 REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]   2> xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]   2> </response>
   [junit4]   2>
   [junit4]   2> request was:q=id:14&qt=standard&start=0&rows=20&version=2.2
   [junit4]   2> 177832 ERROR (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.SolrTestCaseJ4 REQUEST FAILED: q=id:14&qt=standard&start=0&rows=20&version=2.2:java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]   2> xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]   2> </response>
   [junit4]   2>
   [junit4]   2> request was:q=id:14&qt=standard&start=0&rows=20&version=2.2
   [junit4]   2> at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:865)
   [junit4]   2> at org.apache.solr.update.AutoCommitTest.testMaxDocs(AutoCommitTest.java:225)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 177833 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.SolrTestCaseJ4 ###Ending testMaxDocs
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=AutoCommitTest -Dtests.method=testMaxDocs -Dtests.seed=E10E9872A25D54AE -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=fr-CA -Dtests.timezone=Africa/Lome -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   1.34s J2 | AutoCommitTest.testMaxDocs <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Exception during query
   [junit4]    > at __randomizedtesting.SeedInfo.seed([E10E9872A25D54AE:588F4EAD8EB75024]:0)
   [junit4]    > at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:872)
   [junit4]    > at org.apache.solr.update.AutoCommitTest.testMaxDocs(AutoCommitTest.java:225)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]    > xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]    > <response>
   [junit4]    > <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]    > </response>
   [junit4]    > request was:q=id:14&qt=standard&start=0&rows=20&version=2.2
   [junit4]    > at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:865)
   [junit4]    > ... 40 more
   [junit4]   2> 177850 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.SolrTestCaseJ4 ###Starting testCommitWithin
   [junit4]   2> 177850 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 177850 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@659b79ba[collection1] realtime]
   [junit4]   2> 177851 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:* (-9223372036854775807)} 0 0
   [junit4]   2> 177851 ERROR (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.CorePropertiesLocator Couldn't load core descriptor from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/core.properties:java.nio.file.NoSuchFileException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/core.properties
   [junit4]   2> 177851 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 177872 INFO  (commitScheduler-175-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 177873 INFO  (commitScheduler-175-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3709c026[collection1] main]
   [junit4]   2> 177873 INFO  (commitScheduler-175-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 177874 INFO  (searcherExecutor-172-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3709c026[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 177907 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 177955 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 178171 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 178203 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.CoreContainer Reloading SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1
   [junit4]   2> 178216 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@29c3cbc0
   [junit4]   2> 178216 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.update.AutoCommitTest_E10E9872A25D54AE-001/init-core-data-001/]
   [junit4]   2> 178710 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 178710 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 178711 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2beae4a8[collection1] main]
   [junit4]   2> 178712 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 178712 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 178715 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 178718 INFO  (searcherExecutor-176-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 178718 INFO  (searcherExecutor-176-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 178718 INFO  (searcherExecutor-176-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 178718 INFO  (searcherExecutor-176-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 178718 INFO  (searcherExecutor-176-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 178718 INFO  (searcherExecutor-176-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 178719 INFO  (searcherExecutor-176-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 178719 INFO  (searcherExecutor-176-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 178719 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=42, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 178719 INFO  (searcherExecutor-176-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 178720 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 178721 INFO  (searcherExecutor-176-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2beae4a8[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 178722 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@661c38a5[collection1] main]
   [junit4]   2> 178722 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@7b9527f
   [junit4]   2> 178723 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=129585791
   [junit4]   2> 178737 INFO  (searcherExecutor-176-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@661c38a5[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 178756 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[529 (1571152201599942656)]} 0 17
   [junit4]   2> 178757 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:529&qt=standard&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 182750 INFO  (commitScheduler-179-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 182954 INFO  (commitScheduler-179-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@222de3da[collection1] main]
   [junit4]   2> 182960 INFO  (commitScheduler-179-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 182965 INFO  (searcherExecutor-176-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@222de3da[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_5(7.0.0):c1)))}
   [junit4]   2> 183041 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[530 (1571152206104625152)]} 0 4301
   [junit4]   2> 183041 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:530&qt=standard&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 183042 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {delete=[529 (-1571152206108819456)]} 0 4302
   [junit4]   2> 183042 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:529&qt=standard&start=0&rows=20&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 185042 INFO  (commitScheduler-179-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 185118 INFO  (commitScheduler-179-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@23b3dee[collection1] main]
   [junit4]   2> 185118 INFO  (commitScheduler-179-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 185119 INFO  (searcherExecutor-176-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@23b3dee[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_6(7.0.0):c1)))}
   [junit4]   2> 185295 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:529&qt=standard&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 185296 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:530&qt=standard&start=0&rows=20&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 185310 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152208483844096)]} 0 6570
   [junit4]   2> 185310 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152208486989824)]} 0 6570
   [junit4]   2> 185310 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152208486989825)]} 0 6571
   [junit4]   2> 185314 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152208491184128)]} 0 6575
   [junit4]   2> 185315 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152208491184129)]} 0 6575
   [junit4]   2> 185315 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152208492232704)]} 0 6575
   [junit4]   2> 185315 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152208492232705)]} 0 6575
   [junit4]   2> 185315 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152208492232706)]} 0 6576
   [junit4]   2> 185315 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152208492232707)]} 0 6576
   [junit4]   2> 185316 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1571152208493281280)]} 0 6576
   [junit4]   2> 185316 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:500&qt=standard&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 185318 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {delete=[530 (-1571152208494329856)]} 0 6578
   [junit4]   2> 185318 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {delete=[530 (-1571152208495378432)]} 0 6578
   [junit4]   2> 185318 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {delete=[530 (-1571152208495378433)]} 0 6578
   [junit4]   2> 185659 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {delete=[530 (-1571152208495378434)]} 0 6919
   [junit4]   2> 185659 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {delete=[530 (-1571152208852942848)]} 0 6919
   [junit4]   2> 185659 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {delete=[530 (-1571152208852942849)]} 0 6920
   [junit4]   2> 185659 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {delete=[530 (-1571152208852942850)]} 0 6920
   [junit4]   2> 185660 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {delete=[530 (-1571152208852942851)]} 0 6920
   [junit4]   2> 185660 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {delete=[530 (-1571152208853991424)]} 0 6920
   [junit4]   2> 185660 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {delete=[530 (-1571152208853991425)]} 0 6920
   [junit4]   2> 185660 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:530&qt=standard&start=0&rows=20&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 186318 INFO  (commitScheduler-179-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 186331 INFO  (commitScheduler-179-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@c87963a[collection1] main]
   [junit4]   2> 186331 INFO  (commitScheduler-179-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 186332 INFO  (searcherExecutor-176-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@c87963a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_7(7.0.0):c10/9:delGen=1)))}
   [junit4]   2> 186413 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:500&qt=standard&start=0&rows=20&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 186414 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:530&qt=standard&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 186415 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[E10E9872A25D54AE]) [    x:collection1] o.a.s.SolrTestCaseJ4 ###Ending testCommitWithin
   [junit4]   2> 186415 INFO  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    x:collection1] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 186415 INFO  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    x:collection1] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1901875959
   [junit4]   2> 186415 INFO  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 186420 INFO  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 186433 INFO  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 186563 INFO  (SUITE-AutoCommitTest-seed#[E10E9872A25D54AE]-worker) [    x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 186566 INFO  (coreCloseExecutor-180-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4fdb2459
   [junit4]   2> 186566 INFO  (coreCloseExecutor-180-thread-1-processing-x:collection1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter close.
   [junit4]   2> 186566 INFO  (coreCloseExecutor-180-thread-1-processing-x:collection1) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@5f306de0 commitCommandVersion:0
   [junit4]   2> 186618 INFO  (coreCloseExecutor-180-thread-1-processing-x:collection1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1339761753
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.update.AutoCommitTest_E10E9872A25D54AE-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {range_facet_l_dv=PostingsFormat(name=Direct), field_t=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), multiDefault=Lucene50(blocksize=128), subject=Lucene50(blocksize=128), intDefault=PostingsFormat(name=LuceneVarGapDocFreqInterval), id_i1=Lucene50(blocksize=128), id=PostingsFormat(name=Direct), range_facet_i_dv=PostingsFormat(name=LuceneVarGapDocFreqInterval), text=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), range_facet_l=PostingsFormat(name=LuceneVarGapDocFreqInterval), timestamp=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Lucene70), _version_=DocValuesFormat(name=Lucene70), range_facet_i_dv=DocValuesFormat(name=Lucene70), intDvoDefault=DocValuesFormat(name=Direct), timestamp=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=171, maxMBSortInHeap=7.299123566621351, sim=RandomSimilarity(queryNorm=false): {}, locale=fr-CA, timezone=Africa/Lome
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=226093672,total=449839104
   [junit4]   2> NOTE: All tests run in this JVM: [AtomicUpdateProcessorFactoryTest, TestStressInPlaceUpdates, TestReplicationHandlerBackup, TestIndexSearcher, TestMacros, TestFieldCacheSortRandom, AutoCommitTest]
   [junit4] Completed [32/723 (1!)] on J2 in 18.97s, 3 tests, 1 error <<< FAILURES!

[...truncated 219 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsBasicDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZkTest_E10E9872A25D54AE-001/init-core-data-001
   [junit4]   2> 522714 WARN  (SUITE-HdfsBasicDistributedZkTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=28 numCloses=28
   [junit4]   2> 522714 INFO  (SUITE-HdfsBasicDistributedZkTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 522716 INFO  (SUITE-HdfsBasicDistributedZkTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 522716 INFO  (SUITE-HdfsBasicDistributedZkTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 522747 WARN  (SUITE-HdfsBasicDistributedZkTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 522752 WARN  (SUITE-HdfsBasicDistributedZkTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 522754 INFO  (SUITE-HdfsBasicDistributedZkTest-seed#[E10E9872A25D54AE]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 522764 INFO  (SUITE-HdfsBasicDistributedZkTest-seed#[E10E9872A25D54AE]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_37713_hdfs____.nie9wh/webapp
   [junit4]   2> 523100 INFO  (SUITE-HdfsBasicDistributedZkTest-seed#[E10E9872A25D54AE]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37713
   [junit4]   2> 523171 WARN  (SUITE-HdfsBasicDistributedZkTest-seed#[E10E9872A25D54AE]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 523172 INFO  (SUITE-HdfsBasicDistributedZkTest-seed#[E10E9872A25D54AE]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 523182 INFO  (SUITE-HdfsBasicDistributedZkTest-seed#[E10E9872A25D54AE]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_47754_datanode____.3om7za/webapp
   [junit4]   2> 523595 INFO  (SUITE-HdfsBasicDistributedZkTest-seed#[E10E9872A25D54AE]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:47754
   [junit4]   2> 523693 INFO  (IPC Server handler 4 on 46629) [    ] BlockStateChange BLOCK* processReport: from storage DS-e44b3bce-c126-4d76-9c0e-63c5ed033e4e node DatanodeRegistration(127.0.0.1:60303, datanodeUuid=51537665-d2bc-4fa0-91a0-0c3ce9462ea4, infoPort=57721, infoSecurePort=0, ipcPort=43591, storageInfo=lv=-56;cid=testClusterID;nsid=1543253159;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs
   [junit4]   2

[...truncated too long message...]

 1parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 6child3_s=PostingsFormat(name=Memory), 7grand0_s=FSTOrd50, text_t=PostingsFormat(name=LuceneVarGapFixedInterval), 24parent_f2_s=PostingsFormat(name=Memory), 44parent_f2_s=FSTOrd50, 34parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 13parent_f2_s=PostingsFormat(name=Memory), type_s=PostingsFormat(name=LuceneVarGapFixedInterval), 23parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 5child1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 43parent_f2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 10parent_f1_s=FSTOrd50, 33parent_f2_s=FSTOrd50, 11grand1_s=FSTOrd50, 22parent_f2_s=FSTOrd50, 10grand0_s=FSTOrd50, 4child0_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 11grand5_s=FSTOrd50, 12parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 10grand8_s=FSTOrd50, 42parent_f2_s=PostingsFormat(name=Memory), 10grand4_s=FSTOrd50, 32parent_f2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 31parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 5grand1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 6grand4_s=PostingsFormat(name=LuceneVarGapFixedInterval), 11parent_f1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 4grand2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 21parent_f1_s=FSTOrd50, 7grand3_s=PostingsFormat(name=LuceneVarGapFixedInterval), 11grand9_s=FSTOrd50, 8grand2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 9grand5_s=PostingsFormat(name=LuceneVarGapFixedInterval), 8grand6_s=PostingsFormat(name=LuceneVarGapFixedInterval), 9grand1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 41parent_f1_s=PostingsFormat(name=Memory), 37parent_f1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 40parent_f2_s=FSTOrd50, 47parent_f1_s=FSTOrd50, 14parent_f1_s=FSTOrd50, 20parent_f2_s=PostingsFormat(name=Memory), 34parent_f1_s=PostingsFormat(name=Memory), 6child4_s=PostingsFormat(name=LuceneVarGapFixedInterval), 24parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 44parent_f1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 30parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 5parent_f1_s=FSTOrd50, 6grand0_s=PostingsFormat(name=LuceneVarGapFixedInterval), 46parent_f2_s=PostingsFormat(name=Memory), 10parent_f2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 5child0_s=FSTOrd50, 8parent_f1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 27parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 2parent_f2_s=FSTOrd50, 11grand0_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 4child1_s=PostingsFormat(name=Memory), 2grand1_s=FSTOrd50, 11grand4_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 17parent_f2_s=FSTOrd50, 3child2_s=PostingsFormat(name=Memory), 16parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 10grand5_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 26parent_f2_s=FSTOrd50, 1child0_s=PostingsFormat(name=Memory), 36parent_f2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 10grand1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 7grand6_s=PostingsFormat(name=Memory), 5grand0_s=PostingsFormat(name=Memory), 18parent_f1_s=PostingsFormat(name=Memory), 28parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 6grand5_s=PostingsFormat(name=Memory), 7grand2_s=PostingsFormat(name=Memory), 4grand3_s=PostingsFormat(name=Memory), 9grand8_s=FSTOrd50, 15parent_f1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 11grand8_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 8grand3_s=PostingsFormat(name=Memory), 6parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 9grand4_s=FSTOrd50, 25parent_f1_s=FSTOrd50, 8grand7_s=PostingsFormat(name=Memory), 0parent_f1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 3grand2_s=PostingsFormat(name=Memory), 9grand0_s=FSTOrd50, 35parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 4parent_f2_s=PostingsFormat(name=Memory), 19parent_f2_s=PostingsFormat(name=Memory), 37parent_f2_s=FSTOrd50, 47parent_f2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 5child4_s=FSTOrd50, 6child5_s=FSTOrd50, 5grand4_s=PostingsFormat(name=Memory), 38parent_f1_s=PostingsFormat(name=Memory), 6grand1_s=PostingsFormat(name=Memory), 5parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 46parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 6child0_s=PostingsFormat(name=LuceneVarGapFixedInterval), 16parent_f1_s=FSTOrd50, 8parent_f2_s=PostingsFormat(name=Memory), 39parent_f2_s=PostingsFormat(name=Memory), 29parent_f2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 2parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 4child2_s=PostingsFormat(name=LuceneVarGapFixedInterval), rnd_b=PostingsFormat(name=Memory), 11grand3_s=PostingsFormat(name=Memory), 27parent_f1_s=PostingsFormat(name=Memory), 45parent_f1_s=PostingsFormat(name=Memory), 17parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 2grand0_s=PostingsFormat(name=LuceneVarGapFixedInterval), 11grand7_s=PostingsFormat(name=Memory), 3child1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 10grand6_s=PostingsFormat(name=Memory), 7parent_f2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 26parent_f1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 10grand2_s=PostingsFormat(name=Memory), 2child0_s=PostingsFormat(name=LuceneVarGapFixedInterval), 3parent_f1_s=PostingsFormat(name=Memory), 36parent_f1_s=FSTOrd50}, docValues:{_version_=DocValuesFormat(name=Memory), id=DocValuesFormat(name=Lucene70), text_t=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=1327, maxMBSortInHeap=7.260504485524939, sim=RandomSimilarity(queryNorm=false): {}, locale=ar-OM, timezone=Egypt
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=40837008,total=521666560
   [junit4]   2> NOTE: All tests run in this JVM: [AtomicUpdateProcessorFactoryTest, TestStressInPlaceUpdates, TestReplicationHandlerBackup, TestIndexSearcher, TestMacros, TestFieldCacheSortRandom, AutoCommitTest, TestGeoJSONResponseWriter, DistributedSuggestComponentTest, OverseerModifyCollectionTest, TestQueryTypes, TestZkChroot, TestComplexPhraseQParserPlugin, SuggesterFSTTest, SearchHandlerTest, RestartWhileUpdatingTest, TestFastOutputStream, BlockJoinFacetRandomTest, HdfsDirectoryTest, TestBinaryField, TestCloudSchemaless, CoreMergeIndexesAdminHandlerTest, TestReRankQParserPlugin, SuggestComponentContextFilterQueryTest, TestSolrCloudSnapshots, UpdateParamsTest, DirectoryFactoryTest, CdcrReplicationHandlerTest, TestClusterProperties, PropertiesRequestHandlerTest, TestImplicitCoreProperties, TestSerializedLuceneMatchVersion, SolrInfoBeanTest, QueryResultKeyTest, StatsReloadRaceTest, TestRebalanceLeaders, TestMultiValuedNumericRangeQuery, DateRangeFieldTest, ResponseLogComponentTest, TestQueryWrapperFilter, DateMathParserTest, TestSurroundQueryParser, TestSuggestSpellingConverter, TestConfigSetsAPIExclusivity, TestSolrQueryResponse, SaslZkACLProviderTest, TestLegacyFieldCache, TestMiniSolrCloudClusterSSL, TestCorePropertiesReload, RulesTest, SpellCheckCollatorWithCollapseTest, OverseerTaskQueueTest, TestPushWriter, QueryEqualityTest, BigEndianAscendingWordSerializerTest, TestRequestStatusCollectionAPI, TestJettySolrRunner, DistributedExpandComponentTest, StressHdfsTest, TestUseDocValuesAsStored, BasicAuthStandaloneTest, TestGroupingSearch, SolrMetricReporterTest, FullHLLTest, TestShortCircuitedRequests, TestCodecSupport, CollectionsAPISolrJTest, ConnectionManagerTest, TestPostingsSolrHighlighter, SimpleFacetsTest, DeleteReplicaTest, HdfsTlogReplayBufferedWhileIndexingTest, ShowFileRequestHandlerTest, BlockCacheTest, UtilsToolTest, HdfsChaosMonkeyNothingIsSafeTest, TestCloudNestedDocsSort, RecoveryZkTest, BasicDistributedZk2Test, StatsComponentTest, ExternalFileFieldSortTest, TestSubQueryTransformerCrossCore, TestRecovery, TestDefaultStatsCache, TestDynamicFieldResource, V2StandaloneTest, TestTolerantUpdateProcessorCloud, TestTrackingShardHandlerFactory, TestUtils, XsltUpdateRequestHandlerTest, TestMergePolicyConfig, DocExpirationUpdateProcessorFactoryTest, TestLegacyFieldReuse, TestFileDictionaryLookup, TestFilteredDocIdSet, DistanceUnitsTest, ReplicaListTransformerTest, RAMDirectoryFactoryTest, TestConfigSetImmutable, ChaosMonkeyNothingIsSafeWithPullReplicasTest, TestJmxIntegration, AliasIntegrationTest, TestScoreJoinQPScore, TestOverriddenPrefixQueryForCustomFieldType, SharedFSAutoReplicaFailoverUtilsTest, TestConfigsApi, TestReload, MergeStrategyTest, TestQueryUtils, TestSolr4Spatial2, MoveReplicaTest, SimplePostToolTest, TestBinaryResponseWriter, SuggesterTest, TestSQLHandler, TestInitQParser, DistributedFacetPivotWhiteBoxTest, FullSolrCloudDistribCmdsTest]
   [junit4] Completed [441/723 (4!)] on J2 in 2186.54s, 1 test, 1 error <<< FAILURES!

[...truncated 962 lines...]
   [junit4] JVM J2: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20170625_050840_9672767227006505942816.sysout
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps/java_pid8101.hprof ...
   [junit4] Heap dump file created [647524415 bytes in 3.953 secs]
   [junit4] <<< JVM J2: EOF ----

[...truncated 7485 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:818: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:762: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid8101.hprof

Total time: 249 minutes 42 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1337 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1337/

5 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([E8C9B63D7B88AFF9:609D89E7D574C201]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test(ChaosMonkeyNothingIsSafeWithPullReplicasTest.java:208)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest

Error Message:
11 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest:     1) Thread[id=25333, name=StoppableSearchThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:65)    2) Thread[id=25489, name=zkCallback-2392-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=25440, name=StoppableCommitThread-SendThread(127.0.0.1:43397), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    4) Thread[id=25442, name=zkCallback-2392-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=25441, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    6) Thread[id=25487, name=zkCallback-2392-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=25334, name=StoppableCommitThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)    8) Thread[id=25330, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    9) Thread[id=25332, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)   10) Thread[id=25488, name=zkCallback-2392-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)   11) Thread[id=25331, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 11 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest:
   1) Thread[id=25333, name=StoppableSearchThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:65)
   2) Thread[id=25489, name=zkCallback-2392-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=25440, name=StoppableCommitThread-SendThread(127.0.0.1:43397), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   4) Thread[id=25442, name=zkCallback-2392-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=25441, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   6) Thread[id=25487, name=zkCallback-2392-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=25334, name=StoppableCommitThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)
   8) Thread[id=25330, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   9) Thread[id=25332, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
  10) Thread[id=25488, name=zkCallback-2392-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
  11) Thread[id=25331, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
        at __randomizedtesting.SeedInfo.seed([E8C9B63D7B88AFF9]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=25440, name=StoppableCommitThread-SendThread(127.0.0.1:43397), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    2) Thread[id=25442, name=zkCallback-2392-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=25333, name=StoppableSearchThread, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at org.apache.log4j.pattern.BridgePatternConverter.format(BridgePatternConverter.java:119)         at org.apache.log4j.EnhancedPatternLayout.format(EnhancedPatternLayout.java:546)         at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)         at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)         at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)         at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)         at org.apache.log4j.Category.callAppenders(Category.java:206)         at org.apache.log4j.Category.forcedLog(Category.java:391)         at org.apache.log4j.Category.log(Category.java:856)         at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:540)         at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:878)         at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:774)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)         at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)         at org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:54)    4) Thread[id=25489, name=zkCallback-2392-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=25487, name=zkCallback-2392-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=25330, name=StoppableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.io.PrintStream.println(PrintStream.java:805)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)    7) Thread[id=25332, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.io.PrintStream.write(PrintStream.java:478)         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)         at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)         at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)         at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)         at java.io.PrintStream.write(PrintStream.java:527)         at java.io.PrintStream.print(PrintStream.java:669)         at java.io.PrintStream.println(PrintStream.java:823)         at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)         at java.lang.Throwable.printStackTrace(Throwable.java:658)         at java.lang.Throwable.printStackTrace(Throwable.java:643)         at java.lang.Throwable.printStackTrace(Throwable.java:634)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:121)    8) Thread[id=25488, name=zkCallback-2392-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    9) Thread[id=25331, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:379)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:239)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=25440, name=StoppableCommitThread-SendThread(127.0.0.1:43397), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   2) Thread[id=25442, name=zkCallback-2392-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=25333, name=StoppableSearchThread, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at org.apache.log4j.pattern.BridgePatternConverter.format(BridgePatternConverter.java:119)
        at org.apache.log4j.EnhancedPatternLayout.format(EnhancedPatternLayout.java:546)
        at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)
        at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
        at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
        at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
        at org.apache.log4j.Category.callAppenders(Category.java:206)
        at org.apache.log4j.Category.forcedLog(Category.java:391)
        at org.apache.log4j.Category.log(Category.java:856)
        at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:540)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:878)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:774)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
        at org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:54)
   4) Thread[id=25489, name=zkCallback-2392-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=25487, name=zkCallback-2392-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=25330, name=StoppableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.io.PrintStream.println(PrintStream.java:805)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)
   7) Thread[id=25332, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.io.PrintStream.write(PrintStream.java:478)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
        at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
        at java.io.PrintStream.write(PrintStream.java:527)
        at java.io.PrintStream.print(PrintStream.java:669)
        at java.io.PrintStream.println(PrintStream.java:823)
        at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)
        at java.lang.Throwable.printStackTrace(Throwable.java:658)
        at java.lang.Throwable.printStackTrace(Throwable.java:643)
        at java.lang.Throwable.printStackTrace(Throwable.java:634)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:121)
   8) Thread[id=25488, name=zkCallback-2392-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   9) Thread[id=25331, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
        at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:379)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:239)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
        at __randomizedtesting.SeedInfo.seed([E8C9B63D7B88AFF9]:0)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([E8C9B63D7B88AFF9:609D89E7D574C201]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([E8C9B63D7B88AFF9:609D89E7D574C201]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12213 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/init-core-data-001
   [junit4]   2> 1902563 WARN  (SUITE-ChaosMonkeySafeLeaderTest-seed#[E8C9B63D7B88AFF9]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=5 numCloses=5
   [junit4]   2> 1902563 INFO  (SUITE-ChaosMonkeySafeLeaderTest-seed#[E8C9B63D7B88AFF9]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1902564 INFO  (SUITE-ChaosMonkeySafeLeaderTest-seed#[E8C9B63D7B88AFF9]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1902565 INFO  (SUITE-ChaosMonkeySafeLeaderTest-seed#[E8C9B63D7B88AFF9]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /_t/im
   [junit4]   2> 1902568 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1902569 INFO  (Thread-18842) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1902569 INFO  (Thread-18842) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1902578 ERROR (Thread-18842) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1902669 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.ZkTestServer start zk server on port:41241
   [junit4]   2> 1902697 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1902703 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1902709 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1902711 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1902715 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1902718 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1902720 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1902720 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1902721 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1902722 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1902723 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1902730 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 1902871 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1902873 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@75def916{/_t/im,null,AVAILABLE}
   [junit4]   2> 1902874 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@12f022db{SSL,[ssl, http/1.1]}{127.0.0.1:52622}
   [junit4]   2> 1902874 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.e.j.s.Server Started @1905095ms
   [junit4]   2> 1902874 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/tempDir-001/control/data, replicaType=NRT, hostContext=/_t/im, hostPort=52622, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/control-001/cores}
   [junit4]   2> 1902874 ERROR (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1902874 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1902874 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1902874 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1902874 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-26T04:58:37.656Z
   [junit4]   2> 1902881 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1902881 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/control-001/solr.xml
   [junit4]   2> 1902887 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@363cd6e7, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1902891 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41241/solr
   [junit4]   2> 1902922 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1902923 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:52622__t%2Fim
   [junit4]   2> 1902928 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.Overseer Overseer (id=98202623498715141-127.0.0.1:52622__t%2Fim-n_0000000000) starting
   [junit4]   2> 1902941 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52622__t%2Fim
   [junit4]   2> 1902945 INFO  (OverseerStateUpdate-98202623498715141-127.0.0.1:52622__t%2Fim-n_0000000000) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1903016 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@363cd6e7
   [junit4]   2> 1903023 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@363cd6e7
   [junit4]   2> 1903023 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@363cd6e7
   [junit4]   2> 1903028 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/control-001/cores
   [junit4]   2> 1903053 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1903055 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41241/solr ready
   [junit4]   2> 1903081 INFO  (qtp103475315-31155) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&createNodeSet=127.0.0.1:52622__t%252Fim&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1903086 INFO  (OverseerThreadFactory-5486-thread-1-processing-n:127.0.0.1:52622__t%2Fim) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 1903087 INFO  (OverseerThreadFactory-5486-thread-1-processing-n:127.0.0.1:52622__t%2Fim) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.CreateCollectionCmd Only one config set found in zk - using it:conf1
   [junit4]   2> 1903201 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1903201 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1903306 INFO  (zkCallback-2854-thread-1-processing-n:127.0.0.1:52622__t%2Fim) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1904233 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1904253 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 1904386 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1904424 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 1904425 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@363cd6e7
   [junit4]   2> 1904425 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1904425 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 1904428 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@79b80d20
   [junit4]   2> 1904437 WARN  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1904498 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1904498 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1904500 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
   [junit4]   2> 1904500 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1904501 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=50, maxMergedSegmentMB=34.2353515625, floorSegmentMB=1.0224609375, forceMergeDeletesPctAllowed=17.54394000624754, segmentsPerTier=22.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6738114659808392
   [junit4]   2> 1904502 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@8318287[control_collection_shard1_replica_n1] main]
   [junit4]   2> 1904503 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1904504 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1904504 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1904504 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1571241978008436736
   [junit4]   2> 1904507 INFO  (searcherExecutor-5489-thread-1-processing-n:127.0.0.1:52622__t%2Fim x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@8318287[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1904511 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1904511 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1904511 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:52622/_t/im/control_collection_shard1_replica_n1/
   [junit4]   2> 1904511 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1904511 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:52622/_t/im/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 1904511 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1904520 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:52622/_t/im/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 1904621 INFO  (zkCallback-2854-thread-1-processing-n:127.0.0.1:52622__t%2Fim) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1904672 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1904674 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1473
   [junit4]   2> 1904677 INFO  (qtp103475315-31155) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1904775 INFO  (zkCallback-2854-thread-1-processing-n:127.0.0.1:52622__t%2Fim) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1905089 INFO  (OverseerCollectionConfigSetProcessor-98202623498715141-127.0.0.1:52622__t%2Fim-n_0000000000) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1905678 INFO  (qtp103475315-31155) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&createNodeSet=127.0.0.1:52622__t%252Fim&wt=javabin&version=2} status=0 QTime=2596
   [junit4]   2> 1905687 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1905688 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41241/solr ready
   [junit4]   2> 1905688 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1905690 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=1&name=collection1&nrtReplicas=1&action=CREATE&numShards=5&tlogReplicas=0&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1905694 INFO  (OverseerThreadFactory-5486-thread-2-processing-n:127.0.0.1:52622__t%2Fim) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 1905695 INFO  (OverseerThreadFactory-5486-thread-2-processing-n:127.0.0.1:52622__t%2Fim) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.CreateCollectionCmd Only one config set found in zk - using it:conf1
   [junit4]   2> 1905696 WARN  (OverseerThreadFactory-5486-thread-2-processing-n:127.0.0.1:52622__t%2Fim) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 1905902 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1905902 INFO  (qtp103475315-31151) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=1&name=collection1&nrtReplicas=1&action=CREATE&numShards=5&tlogReplicas=0&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=211
   [junit4]   2> 1906024 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/shard-1-001 of type NRT
   [junit4]   2> 1906026 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1906027 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5af455e6{/_t/im,null,AVAILABLE}
   [junit4]   2> 1906027 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@40445025{SSL,[ssl, http/1.1]}{127.0.0.1:33741}
   [junit4]   2> 1906027 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.e.j.s.Server Started @1908248ms
   [junit4]   2> 1906027 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/_t/im, hostPort=33741, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/shard-1-001/cores}
   [junit4]   2> 1906028 ERROR (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1906028 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1906028 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1906028 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1906028 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-26T04:58:40.810Z
   [junit4]   2> 1906053 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1906053 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/shard-1-001/solr.xml
   [junit4]   2> 1906057 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@363cd6e7, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1906061 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41241/solr
   [junit4]   2> 1906082 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:33741__t%2Fim    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1906085 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:33741__t%2Fim    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1906087 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:33741__t%2Fim    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33741__t%2Fim
   [junit4]   2> 1906088 INFO  (zkCallback-2854-thread-2-processing-n:127.0.0.1:52622__t%2Fim) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1906088 INFO  (zkCallback-2861-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1906089 INFO  (zkCallback-2866-thread-1-processing-n:127.0.0.1:33741__t%2Fim) [n:127.0.0.1:33741__t%2Fim    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1906152 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:33741__t%2Fim    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@363cd6e7
   [junit4]   2> 1906157 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:33741__t%2Fim    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@363cd6e7
   [junit4]   2> 1906157 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:33741__t%2Fim    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@363cd6e7
   [junit4]   2> 1906158 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:33741__t%2Fim    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/shard-1-001/cores
   [junit4]   2> 1906176 INFO  (qtp103475315-31155) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:33741__t%252Fim&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1906179 INFO  (OverseerCollectionConfigSetProcessor-98202623498715141-127.0.0.1:52622__t%2Fim-n_0000000000) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1906179 INFO  (OverseerThreadFactory-5486-thread-3-processing-n:127.0.0.1:52622__t%2Fim) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.AddReplicaCmd addReplica() : {
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "node":"127.0.0.1:33741__t%2Fim",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"addreplica"}
   [junit4]   2> 1906179 INFO  (OverseerThreadFactory-5486-thread-3-processing-n:127.0.0.1:52622__t%2Fim) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:33741__t%2Fim for creating new replica
   [junit4]   2> 1906201 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1906202 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1907216 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1907231 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema [collection1_shard2_replica_n0] Schema name=test
   [junit4]   2> 1907348 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1907378 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n0' using configuration from collection collection1, trusted=true
   [junit4]   2> 1907378 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n0' (registry 'solr.core.collection1.shard2.replica_n0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@363cd6e7
   [junit4]   2> 1907379 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1907379 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore [[collection1_shard2_replica_n0] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/shard-1-001/cores/collection1_shard2_replica_n0], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/shard-1-001/cores/collection1_shard2_replica_n0/data/]
   [junit4]   2> 1907382 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@106560ed
   [junit4]   2> 1907391 WARN  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1907441 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1907441 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1907442 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
   [junit4]   2> 1907442 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1907443 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=50, maxMergedSegmentMB=34.2353515625, floorSegmentMB=1.0224609375, forceMergeDeletesPctAllowed=17.54394000624754, segmentsPerTier=22.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6738114659808392
   [junit4]   2> 1907444 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.SolrIndexSearcher Opening [Searcher@4276bd3[collection1_shard2_replica_n0] main]
   [junit4]   2> 1907446 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1907447 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1907448 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1907448 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1571241981095444480
   [junit4]   2> 1907451 INFO  (searcherExecutor-5500-thread-1-processing-n:127.0.0.1:33741__t%2Fim x:collection1_shard2_replica_n0 s:shard2 c:collection1) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore [collection1_shard2_replica_n0] Registered new searcher Searcher@4276bd3[collection1_shard2_replica_n0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1907471 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1907471 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1907471 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33741/_t/im/collection1_shard2_replica_n0/
   [junit4]   2> 1907471 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1907471 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy https://127.0.0.1:33741/_t/im/collection1_shard2_replica_n0/ has no replicas
   [junit4]   2> 1907471 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1907476 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:33741/_t/im/collection1_shard2_replica_n0/ shard2
   [junit4]   2> 1907627 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1907629 INFO  (qtp618930097-31197) [n:127.0.0.1:33741__t%2Fim c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1427
   [junit4]   2> 1907630 INFO  (qtp103475315-31155) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:33741__t%252Fim&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1453
   [junit4]   2> 1907753 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/shard-2-001 of type NRT
   [junit4]   2> 1907755 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1907758 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@18522160{/_t/im,null,AVAILABLE}
   [junit4]   2> 1907759 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7e52a0e{SSL,[ssl, http/1.1]}{127.0.0.1:46745}
   [junit4]   2> 1907759 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.e.j.s.Server Started @1909979ms
   [junit4]   2> 1907759 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/_t/im, hostPort=46745, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/shard-2-001/cores}
   [junit4]   2> 1907759 ERROR (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1907760 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1907760 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1907760 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1907760 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-26T04:58:42.542Z
   [junit4]   2> 1907771 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1907771 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001/shard-2-001/solr.xml
   [junit4]   2> 1907782 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@363cd6e7, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1907786 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41241/solr
   [junit4]   2> 1907809 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:46745__t%2Fim    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1907814 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:46745__t%2Fim    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1907820 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[E8C9B63D7B88AFF9]) [n:127.0.0.1:46745__t%2Fim    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46745__t%2Fim
   [junit4]   2> 1907821 INFO  (zkCallback-2854-thread-2-processing-n:127.0.0.1:52622__t%2Fim) [n:127.0.0.1:52622__t%2Fim    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1907821 INFO  (zkCallback-2861-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1907821 INFO  (zkCallback-2866-thread-1-processing-n:127.0.0.1:33741__t%2Fim) [n:127.0.0.1:33741__t%2Fim    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1907833 INFO  (zkCallback-2872-thread-1-pro

[...truncated too long message...]

invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MethodMetric$2.snapshot(MethodMetric.java:111)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MethodMetric.snapshot(MethodMetric.java:144)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MetricsRegistry.snapshot(MetricsRegistry.java:401)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder$1.getMetrics(MetricsSourceBuilder.java:79)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:194)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1804)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1595)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2> at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2> at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:226)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:204)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest.teardownClass(HdfsChaosMonkeySafeLeaderTest.java:47)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> ... 54 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_E8C9B63D7B88AFF9-001
   [junit4]   2> Jun 26, 2017 5:27:35 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 35 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {rnd_b=PostingsFormat(name=LuceneVarGapFixedInterval), a_t=PostingsFormat(name=Memory), a_i=PostingsFormat(name=LuceneVarGapFixedInterval), id=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{rnd_b=DocValuesFormat(name=Memory), _version_=DocValuesFormat(name=Lucene70), a_t=DocValuesFormat(name=Lucene70), a_i=DocValuesFormat(name=Memory), id=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=1322, maxMBSortInHeap=5.1910971334816525, sim=RandomSimilarity(queryNorm=false): {}, locale=sr-RS, timezone=Etc/GMT+4
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=2,free=267239456,total=529006592
   [junit4]   2> NOTE: All tests run in this JVM: [TestXmlQParserPlugin, OpenExchangeRatesOrgProviderTest, FacetPivotSmallTest, TestReqParamsAPI, TestStressLiveNodes, TestInPlaceUpdatesDistrib, TestFastLRUCache, SpellCheckCollatorTest, TestFieldCacheReopen, TermVectorComponentTest, AddBlockUpdateTest, TestMinMaxOnMultiValuedField, MetricUtilsTest, HighlighterConfigTest, TestCSVLoader, ShardSplitTest, TestCustomDocTransformer, SolrIndexSplitterTest, HdfsSyncSliceTest, PrimitiveFieldTypeTest, TestRemoteStreaming, VersionInfoTest, TestSubQueryTransformerDistrib, BasicDistributedZkTest, TestSolrDeletionPolicy1, TestJoin, DistributedQueryComponentCustomSortTest, TestReplicaProperties, RequestHandlersTest, CursorMarkTest, TestPayloadCheckQParserPlugin, TestBackupRepositoryFactory, ReplaceNodeTest, TestLegacyNumericRangeQueryBuilder, TestFastWriter, TestDistributedSearch, DistribDocExpirationUpdateProcessorTest, ShufflingReplicaListTransformerTest, TestCopyFieldCollectionResource, TestApiFramework, NotRequiredUniqueKeyTest, TestSolrCoreSnapshots, MultiThreadedOCPTest, QueryElevationComponentTest, CdcrRequestHandlerTest, TestSSLRandomization, TestRandomFlRTGCloud, TestDocSet, TestDistribIDF, FieldMutatingUpdateProcessorTest, PingRequestHandlerTest, CoreAdminCreateDiscoverTest, TestCloudPseudoReturnFields, TestGraphMLResponseWriter, ResponseHeaderTest, RankQueryTest, TolerantUpdateProcessorTest, DeleteShardTest, TestSQLHandlerNonCloud, ExitableDirectoryReaderTest, SolrIndexConfigTest, MigrateRouteKeyTest, SolrCoreMetricManagerTest, SynonymTokenizerTest, TestPerFieldSimilarity, TestEmbeddedSolrServerConstructors, ConnectionReuseTest, TestHdfsCloudBackupRestore, TestManagedSchemaAPI, CloudExitableDirectoryReaderTest, TestNestedDocsSort, TestReversedWildcardFilterFactory, TestCollationField, TestCollationFieldDocValues, TestExtendedDismaxParser, TestCloudManagedSchema, WordBreakSolrSpellCheckerTest, DefaultValueUpdateProcessorTest, DistributedFacetExistsSmallTest, TestCustomStream, HLLSerializationTest, TestNumericTerms32, PeerSyncReplicationTest, SOLR749Test, TestCoreAdminApis, IndexBasedSpellCheckerTest, TestSolr4Spatial, TestRangeQuery, CreateCollectionCleanupTest, SolrRequestParserTest, TestAnalyzedSuggestions, DebugComponentTest, TestIntervalFaceting, TestClassicSimilarityFactory, DocValuesMissingTest, TestBadConfig, ImplicitSnitchTest, InfoHandlerTest, TestRestManager, TestSolrCloudWithSecureImpersonation, TestMaxTokenLenTokenizer, SimpleCollectionCreateDeleteTest, HttpPartitionTest, TestSimpleQParserPlugin, HdfsChaosMonkeySafeLeaderTest]
   [junit4] Completed [711/723 (3!)] on J0 in 84.36s, 1 test, 1 failure <<< FAILURES!

[...truncated 43240 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1338 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1338/

5 tests failed.
FAILED:  org.apache.lucene.index.TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreadsOnlyOnce

Error Message:
hit unexpected Throwable

Stack Trace:
java.lang.AssertionError: hit unexpected Throwable
        at __randomizedtesting.SeedInfo.seed([66484822FD0371B5:2A23C9E43BA35250]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.lucene.index.TestIndexWriterWithThreads._testMultipleThreadsFailure(TestIndexWriterWithThreads.java:300)
        at org.apache.lucene.index.TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreadsOnlyOnce(TestIndexWriterWithThreads.java:497)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.CollectionsAPISolrJTest.testCreateAndDeleteShard

Error Message:
Error from server at https://127.0.0.1:57457/solr: deleteshard the collection time out:180s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:57457/solr: deleteshard the collection time out:180s
        at __randomizedtesting.SeedInfo.seed([CD4BE89D12168533:8155C8194E34FCE]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:624)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:239)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:470)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:400)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1102)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:843)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:774)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.CollectionsAPISolrJTest.testCreateAndDeleteShard(CollectionsAPISolrJTest.java:143)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestPullReplica.testKillLeader

Error Message:
Replica state not updated in cluster state null Live Nodes: [127.0.0.1:40238_solr, 127.0.0.1:43485_solr] Last available state: DocCollection(pull_replica_test_kill_leader//collections/pull_replica_test_kill_leader/state.json/6)={   "pullReplicas":"1",   "replicationFactor":"1",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "core":"pull_replica_test_kill_leader_shard1_replica_n1",           "base_url":"http://127.0.0.1:43485/solr",           "node_name":"127.0.0.1:43485_solr",           "state":"down",           "type":"NRT",           "leader":"true"},         "core_node2":{           "core":"pull_replica_test_kill_leader_shard1_replica_p1",           "base_url":"http://127.0.0.1:40238/solr",           "node_name":"127.0.0.1:40238_solr",           "state":"active",           "type":"PULL"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"100",   "autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Replica state not updated in cluster state
null
Live Nodes: [127.0.0.1:40238_solr, 127.0.0.1:43485_solr]
Last available state: DocCollection(pull_replica_test_kill_leader//collections/pull_replica_test_kill_leader/state.json/6)={
  "pullReplicas":"1",
  "replicationFactor":"1",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "core":"pull_replica_test_kill_leader_shard1_replica_n1",
          "base_url":"http://127.0.0.1:43485/solr",
          "node_name":"127.0.0.1:43485_solr",
          "state":"down",
          "type":"NRT",
          "leader":"true"},
        "core_node2":{
          "core":"pull_replica_test_kill_leader_shard1_replica_p1",
          "base_url":"http://127.0.0.1:40238/solr",
          "node_name":"127.0.0.1:40238_solr",
          "state":"active",
          "type":"PULL"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"100",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([CD4BE89D12168533:845D1C2970AD1165]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:269)
        at org.apache.solr.cloud.TestPullReplica.doTestNoLeader(TestPullReplica.java:401)
        at org.apache.solr.cloud.TestPullReplica.testKillLeader(TestPullReplica.java:290)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv

Error Message:
java.lang.RuntimeException: Error from server at http://127.0.0.1:46954/solr/test_col: Failed synchronous update on shard StdNode: http://127.0.0.1:39465/solr/test_col_shard3_replica_n1/ update: org.apache.solr.client.solrj.request.UpdateRequest@49b8f9d2

Stack Trace:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error from server at http://127.0.0.1:46954/solr/test_col: Failed synchronous update on shard StdNode: http://127.0.0.1:39465/solr/test_col_shard3_replica_n1/ update: org.apache.solr.client.solrj.request.UpdateRequest@49b8f9d2
        at __randomizedtesting.SeedInfo.seed([CD4BE89D12168533:FB5F8ADB984BBF22]:0)
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:281)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv(TestStressCloudBlindAtomicUpdates.java:193)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Error from server at http://127.0.0.1:46954/solr/test_col: Failed synchronous update on shard StdNode: http://127.0.0.1:39465/solr/test_col_shard3_replica_n1/ update: org.apache.solr.client.solrj.request.UpdateRequest@49b8f9d2
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.run(TestStressCloudBlindAtomicUpdates.java:409)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        ... 1 more
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:46954/solr/test_col: Failed synchronous update on shard StdNode: http://127.0.0.1:39465/solr/test_col_shard3_replica_n1/ update: org.apache.solr.client.solrj.request.UpdateRequest@49b8f9d2
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:624)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:239)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.doRandomAtomicUpdate(TestStressCloudBlindAtomicUpdates.java:368)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.run(TestStressCloudBlindAtomicUpdates.java:397)
        ... 6 more


FAILED:  org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig

Error Message:
The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.

Stack Trace:
java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
        at __randomizedtesting.SeedInfo.seed([CD4BE89D12168533:3AE421B6EF9F6F18]:0)
        at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:311)
        at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:287)
        at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:227)
        at org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig(HdfsDirectoryFactoryTest.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:693)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
        at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:71)
        at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:309)
        ... 42 more




Build Log:
[...truncated 858 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterWithThreads
   [junit4]   1> Thread-12393: ERROR: unexpected Throwable:
   [junit4]   1> java.lang.AssertionError: numTerms2=0 vs -8
   [junit4]   1> at org.apache.lucene.index.BufferedUpdatesStream.checkDeleteStats(BufferedUpdatesStream.java:376)
   [junit4]   1> at org.apache.lucene.index.BufferedUpdatesStream.push(BufferedUpdatesStream.java:85)
   [junit4