[JENKINS] Lucene-Solr-Tests-master - Build # 2101 - Still Unstable

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-master - Build # 2101 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/2101/

3 tests failed.
FAILED:  org.apache.solr.cloud.PeerSyncReplicationTest.test

Error Message:
expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: expected:<0> but was:<1>
        at __randomizedtesting.SeedInfo.seed([82F6963EC18A4E82:AA2A9E46F76237A]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.cloud.PeerSyncReplicationTest.test(PeerSyncReplicationTest.java:202)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
Didn't see all replicas for shard shard1 in collMinRf_1x3 come up within 90000 ms! ClusterState: {   "control_collection":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{"shard1":{         "range":"80000000-7fffffff",         "state":"active",         "replicas":{"core_node2":{             "core":"control_collection_shard1_replica_n1",             "base_url":"http://127.0.0.1:40260",             "node_name":"127.0.0.1:40260_",             "state":"active",             "type":"NRT",             "leader":"true"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"1",     "tlogReplicas":"0"},   "collMinRf_1x3":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{"shard1":{         "range":"80000000-7fffffff",         "state":"active",         "replicas":{           "core_node4":{             "core":"collMinRf_1x3_shard1_replica_t2",             "base_url":"http://127.0.0.1:34062",             "node_name":"127.0.0.1:34062_",             "state":"active",             "type":"TLOG"},           "core_node5":{             "core":"collMinRf_1x3_shard1_replica_t1",             "base_url":"http://127.0.0.1:34862",             "node_name":"127.0.0.1:34862_",             "state":"active",             "type":"TLOG",             "leader":"true"},           "core_node6":{             "core":"collMinRf_1x3_shard1_replica_t3",             "base_url":"http://127.0.0.1:40260",             "node_name":"127.0.0.1:40260_",             "state":"recovering",             "type":"TLOG"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"0",     "tlogReplicas":"3"},   "collection1":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{       "shard1":{         "range":"80000000-ffffffff",         "state":"active",         "replicas":{"core_node44":{             "core":"collection1_shard1_replica_t43",             "base_url":"http://127.0.0.1:34862",             "node_name":"127.0.0.1:34862_",             "state":"active",             "type":"TLOG",             "leader":"true"}}},       "shard2":{         "range":"0-7fffffff",         "state":"active",         "replicas":{           "core_node42":{             "core":"collection1_shard2_replica_t41",             "base_url":"http://127.0.0.1:34062",             "node_name":"127.0.0.1:34062_",             "state":"active",             "type":"TLOG",             "leader":"true"},           "core_node46":{             "core":"collection1_shard2_replica_t45",             "base_url":"http://127.0.0.1:39486",             "node_name":"127.0.0.1:39486_",             "state":"active",             "type":"TLOG"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"1",     "tlogReplicas":"0"}}

Stack Trace:
java.lang.AssertionError: Didn't see all replicas for shard shard1 in collMinRf_1x3 come up within 90000 ms! ClusterState: {
  "control_collection":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{"core_node2":{
            "core":"control_collection_shard1_replica_n1",
            "base_url":"http://127.0.0.1:40260",
            "node_name":"127.0.0.1:40260_",
            "state":"active",
            "type":"NRT",
            "leader":"true"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"1",
    "tlogReplicas":"0"},
  "collMinRf_1x3":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{
          "core_node4":{
            "core":"collMinRf_1x3_shard1_replica_t2",
            "base_url":"http://127.0.0.1:34062",
            "node_name":"127.0.0.1:34062_",
            "state":"active",
            "type":"TLOG"},
          "core_node5":{
            "core":"collMinRf_1x3_shard1_replica_t1",
            "base_url":"http://127.0.0.1:34862",
            "node_name":"127.0.0.1:34862_",
            "state":"active",
            "type":"TLOG",
            "leader":"true"},
          "core_node6":{
            "core":"collMinRf_1x3_shard1_replica_t3",
            "base_url":"http://127.0.0.1:40260",
            "node_name":"127.0.0.1:40260_",
            "state":"recovering",
            "type":"TLOG"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"0",
    "tlogReplicas":"3"},
  "collection1":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{
      "shard1":{
        "range":"80000000-ffffffff",
        "state":"active",
        "replicas":{"core_node44":{
            "core":"collection1_shard1_replica_t43",
            "base_url":"http://127.0.0.1:34862",
            "node_name":"127.0.0.1:34862_",
            "state":"active",
            "type":"TLOG",
            "leader":"true"}}},
      "shard2":{
        "range":"0-7fffffff",
        "state":"active",
        "replicas":{
          "core_node42":{
            "core":"collection1_shard2_replica_t41",
            "base_url":"http://127.0.0.1:34062",
            "node_name":"127.0.0.1:34062_",
            "state":"active",
            "type":"TLOG",
            "leader":"true"},
          "core_node46":{
            "core":"collection1_shard2_replica_t45",
            "base_url":"http://127.0.0.1:39486",
            "node_name":"127.0.0.1:39486_",
            "state":"active",
            "type":"TLOG"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"1",
    "tlogReplicas":"0"}}
        at __randomizedtesting.SeedInfo.seed([82F6963EC18A4E82:AA2A9E46F76237A]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.ensureAllReplicasAreActive(AbstractFullDistribZkTestBase.java:1996)
        at org.apache.solr.cloud.HttpPartitionTest.testMinRf(HttpPartitionTest.java:248)
        at org.apache.solr.cloud.HttpPartitionTest.test(HttpPartitionTest.java:127)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestCloudRecovery.corruptedLogTest

Error Message:
Timeout waiting for all live and active

Stack Trace:
java.lang.AssertionError: Timeout waiting for all live and active
        at __randomizedtesting.SeedInfo.seed([82F6963EC18A4E82:180C9CC17F34023]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.TestCloudRecovery.corruptedLogTest(TestCloudRecovery.java:184)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12002 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/init-core-data-001
   [junit4]   2> 2068323 WARN  (SUITE-HttpPartitionTest-seed#[82F6963EC18A4E82]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=148 numCloses=148
   [junit4]   2> 2068344 INFO  (SUITE-HttpPartitionTest-seed#[82F6963EC18A4E82]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2068345 INFO  (SUITE-HttpPartitionTest-seed#[82F6963EC18A4E82]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 2068345 INFO  (SUITE-HttpPartitionTest-seed#[82F6963EC18A4E82]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2068345 INFO  (SUITE-HttpPartitionTest-seed#[82F6963EC18A4E82]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 2068346 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2068354 INFO  (Thread-1765) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2068354 INFO  (Thread-1765) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2068497 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.ZkTestServer start zk server on port:40266
   [junit4]   2> 2068628 ERROR (Thread-1765) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2069064 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2069122 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2069123 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2069124 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2069125 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2069138 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2069138 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2069139 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2069140 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2069140 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2069141 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2069213 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use TLOG replicas unless explicitly asked otherwise
   [junit4]   2> 2069214 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15e90da9e760001, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2070151 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2070210 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@bf868d2{/,null,AVAILABLE}
   [junit4]   2> 2070211 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4caff25f{HTTP/1.1,[http/1.1]}{127.0.0.1:45057}
   [junit4]   2> 2070211 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.e.j.s.Server Started @2084154ms
   [junit4]   2> 2070211 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=40260, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/control-001/cores}
   [junit4]   2> 2070211 ERROR (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2070211 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2070211 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2070211 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 2070211 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-17T17:20:01.934Z
   [junit4]   2> 2070312 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2070312 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/control-001/solr.xml
   [junit4]   2> 2070319 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15e90da9e760002, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2070328 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2070364 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40266/solr
   [junit4]   2> 2070466 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15e90da9e760003, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2071685 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:40260_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2071686 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:40260_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:40260_
   [junit4]   2> 2071687 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:40260_    ] o.a.s.c.Overseer Overseer (id=98675510484533252-127.0.0.1:40260_-n_0000000000) starting
   [junit4]   2> 2071810 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:40260_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40260_
   [junit4]   2> 2071811 INFO  (zkCallback-2061-thread-1-processing-n:127.0.0.1:40260_) [n:127.0.0.1:40260_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2072704 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:40260_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed
   [junit4]   2> 2072778 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:40260_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed
   [junit4]   2> 2072779 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:40260_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed
   [junit4]   2> 2072795 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:40260_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/control-001/cores
   [junit4]   2> 2072922 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2072923 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40266/solr ready
   [junit4]   2> 2072924 INFO  (SocketProxy-Acceptor-40260) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=51968,localport=40260], receiveBufferSize:531000
   [junit4]   2> 2073029 INFO  (SocketProxy-Acceptor-40260) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=45057,localport=36620], receiveBufferSize=530904
   [junit4]   2> 2073065 INFO  (qtp163246312-8168) [n:127.0.0.1:40260_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:40260_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2073100 INFO  (OverseerThreadFactory-2435-thread-1-processing-n:127.0.0.1:40260_) [n:127.0.0.1:40260_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 2073246 INFO  (SocketProxy-Acceptor-40260) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=51972,localport=40260], receiveBufferSize:531000
   [junit4]   2> 2073279 INFO  (SocketProxy-Acceptor-40260) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=45057,localport=36624], receiveBufferSize=530904
   [junit4]   2> 2073299 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2073300 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2073409 INFO  (zkCallback-2061-thread-1-processing-n:127.0.0.1:40260_) [n:127.0.0.1:40260_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2074526 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2074586 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2075672 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2075809 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 2075810 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed
   [junit4]   2> 2075810 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2075810 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 2075813 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8508779157034069]
   [junit4]   2> 2075868 WARN  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2076191 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2076191 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2076192 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2076192 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2076193 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=14.3369140625, floorSegmentMB=1.78515625, forceMergeDeletesPctAllowed=20.927308559770495, segmentsPerTier=49.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2076229 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2838b1ac[control_collection_shard1_replica_n1] main]
   [junit4]   2> 2076284 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2076300 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2076301 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2076301 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578808176042573824
   [junit4]   2> 2076309 INFO  (searcherExecutor-2438-thread-1-processing-n:127.0.0.1:40260_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@2838b1ac[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2076376 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2076376 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2076376 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40260/control_collection_shard1_replica_n1/
   [junit4]   2> 2076376 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2076376 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:40260/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 2076376 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2076394 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40260/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 2076421 INFO  (zkCallback-2061-thread-1-processing-n:127.0.0.1:40260_) [n:127.0.0.1:40260_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2076541 INFO  (zkCallback-2061-thread-2-processing-n:127.0.0.1:40260_) [n:127.0.0.1:40260_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2076556 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2076557 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3258
   [junit4]   2> 2076593 INFO  (qtp163246312-8168) [n:127.0.0.1:40260_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2076731 INFO  (zkCallback-2061-thread-1-processing-n:127.0.0.1:40260_) [n:127.0.0.1:40260_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2077121 INFO  (OverseerCollectionConfigSetProcessor-98675510484533252-127.0.0.1:40260_-n_0000000000) [n:127.0.0.1:40260_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2077597 INFO  (qtp163246312-8168) [n:127.0.0.1:40260_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:40260_&wt=javabin&version=2} status=0 QTime=4531
   [junit4]   2> 2077753 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2077753 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40266/solr ready
   [junit4]   2> 2077753 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2077754 INFO  (SocketProxy-Acceptor-40260) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=51978,localport=40260], receiveBufferSize:531000
   [junit4]   2> 2077779 INFO  (SocketProxy-Acceptor-40260) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=45057,localport=36630], receiveBufferSize=530904
   [junit4]   2> 2077848 INFO  (qtp163246312-8169) [n:127.0.0.1:40260_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2077869 INFO  (OverseerThreadFactory-2435-thread-2-processing-n:127.0.0.1:40260_) [n:127.0.0.1:40260_    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 2077870 WARN  (OverseerThreadFactory-2435-thread-2-processing-n:127.0.0.1:40260_) [n:127.0.0.1:40260_    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 2078104 INFO  (qtp163246312-8169) [n:127.0.0.1:40260_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2078104 INFO  (qtp163246312-8169) [n:127.0.0.1:40260_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=255
   [junit4]   2> 2078652 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/shard-1-001 of type TLOG
   [junit4]   2> 2078652 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2078653 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@717f3c86{/,null,AVAILABLE}
   [junit4]   2> 2078653 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@677715ad{HTTP/1.1,[http/1.1]}{127.0.0.1:38681}
   [junit4]   2> 2078653 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.e.j.s.Server Started @2092596ms
   [junit4]   2> 2078653 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/tempDir-001/jetty1, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=34062, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/shard-1-001/cores}
   [junit4]   2> 2078654 ERROR (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2078654 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2078654 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2078654 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 2078654 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-17T17:20:10.377Z
   [junit4]   2> 2078703 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2078703 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/shard-1-001/solr.xml
   [junit4]   2> 2078813 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2078883 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40266/solr
   [junit4]   2> 2079032 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:34062_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2079065 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:34062_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2079082 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:34062_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34062_
   [junit4]   2> 2079084 INFO  (zkCallback-2068-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2079136 INFO  (zkCallback-2061-thread-1-processing-n:127.0.0.1:40260_) [n:127.0.0.1:40260_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2079198 INFO  (zkCallback-2073-thread-1-processing-n:127.0.0.1:34062_) [n:127.0.0.1:34062_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2079706 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:34062_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed
   [junit4]   2> 2079757 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:34062_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed
   [junit4]   2> 2079757 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:34062_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed
   [junit4]   2> 2079758 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:34062_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/shard-1-001/cores
   [junit4]   2> 2079869 INFO  (OverseerCollectionConfigSetProcessor-98675510484533252-127.0.0.1:40260_-n_0000000000) [n:127.0.0.1:40260_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2080060 INFO  (qtp163246312-8170) [n:127.0.0.1:40260_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:34062_&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2080126 INFO  (OverseerThreadFactory-2435-thread-3-processing-n:127.0.0.1:40260_) [n:127.0.0.1:40260_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:34062_ for creating new replica
   [junit4]   2> 2080141 INFO  (SocketProxy-Acceptor-34062) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=45808,localport=34062], receiveBufferSize:531000
   [junit4]   2> 2080194 INFO  (SocketProxy-Acceptor-34062) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=38681,localport=55616], receiveBufferSize=530904
   [junit4]   2> 2080296 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2080297 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2080333 INFO  (zkCallback-2073-thread-1-processing-n:127.0.0.1:34062_) [n:127.0.0.1:34062_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2081433 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2081545 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.IndexSchema [collection1_shard2_replica_t41] Schema name=test
   [junit4]   2> 2082250 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2082339 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_t41' using configuration from collection collection1, trusted=true
   [junit4]   2> 2082339 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_t41' (registry 'solr.core.collection1.shard2.replica_t41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed
   [junit4]   2> 2082339 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2082339 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SolrCore [[collection1_shard2_replica_t41] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/shard-1-001/cores/collection1_shard2_replica_t41], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/shard-1-001/cores/collection1_shard2_replica_t41/data/]
   [junit4]   2> 2082359 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8508779157034069]
   [junit4]   2> 2082433 WARN  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2082797 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2082797 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2082798 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2082798 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2082799 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=14.3369140625, floorSegmentMB=1.78515625, forceMergeDeletesPctAllowed=20.927308559770495, segmentsPerTier=49.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2082814 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.SolrIndexSearcher Opening [Searcher@f044a2e[collection1_shard2_replica_t41] main]
   [junit4]   2> 2082835 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2082835 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2082835 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2082836 INFO  (searcherExecutor-2449-thread-1-processing-n:127.0.0.1:34062_ x:collection1_shard2_replica_t41 s:shard2 c:collection1) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SolrCore [collection1_shard2_replica_t41] Registered new searcher Searcher@f044a2e[collection1_shard2_replica_t41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2082837 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578808182896066560
   [junit4]   2> 2083016 INFO  (zkCallback-2073-thread-1-processing-n:127.0.0.1:34062_) [n:127.0.0.1:34062_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2083016 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2083016 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2083016 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:34062/collection1_shard2_replica_t41/
   [junit4]   2> 2083016 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2083016 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SyncStrategy http://127.0.0.1:34062/collection1_shard2_replica_t41/ has no replicas
   [junit4]   2> 2083016 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2083017 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ZkController collection1_shard2_replica_t41 stopping background replication from leader
   [junit4]   2> 2083019 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:34062/collection1_shard2_replica_t41/ shard2
   [junit4]   2> 2083120 INFO  (zkCallback-2073-thread-1-processing-n:127.0.0.1:34062_) [n:127.0.0.1:34062_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2083134 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2083135 INFO  (qtp47296689-8217) [n:127.0.0.1:34062_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2838
   [junit4]   2> 2083171 INFO  (qtp163246312-8170) [n:127.0.0.1:40260_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:34062_&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2} status=0 QTime=3111
   [junit4]   2> 2083300 INFO  (zkCallback-2073-thread-1-processing-n:127.0.0.1:34062_) [n:127.0.0.1:34062_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2083958 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/shard-2-001 of type TLOG
   [junit4]   2> 2083958 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2084031 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@469ff462{/,null,AVAILABLE}
   [junit4]   2> 2084031 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@427c9c29{HTTP/1.1,[http/1.1]}{127.0.0.1:33061}
   [junit4]   2> 2084031 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.e.j.s.Server Started @2097974ms
   [junit4]   2> 2084031 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/tempDir-001/jetty2, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=34862, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/shard-2-001/cores}
   [junit4]   2> 2084031 ERROR (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2084081 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2084081 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2084081 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 2084081 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-17T17:20:15.804Z
   [junit4]   2> 2084127 INFO  (OverseerCollectionConfigSetProcessor-98675510484533252-127.0.0.1:40260_-n_0000000000) [n:127.0.0.1:40260_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2084127 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2084127 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/shard-2-001/solr.xml
   [junit4]   2> 2084200 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2084272 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40266/solr
   [junit4]   2> 2084347 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:34862_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2084360 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:34862_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2084364 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:34862_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34862_
   [junit4]   2> 2084385 INFO  (zkCallback-2073-thread-1-processing-n:127.0.0.1:34062_) [n:127.0.0.1:34062_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2084458 INFO  (zkCallback-2068-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2084491 INFO  (zkCallback-2061-thread-1-processing-n:127.0.0.1:40260_) [n:127.0.0.1:40260_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2084543 INFO  (zkCallback-2079-thread-1-processing-n:127.0.0.1:34862_) [n:127.0.0.1:34862_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2084783 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:34862_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed
   [junit4]   2> 2084837 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:34862_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed
   [junit4]   2> 2084837 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:34862_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed
   [junit4]   2> 2084839 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:34862_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/shard-2-001/cores
   [junit4]   2> 2085259 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:34862_&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2085316 INFO  (OverseerThreadFactory-2435-thread-4-processing-n:127.0.0.1:40260_) [n:127.0.0.1:40260_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:34862_ for creating new replica
   [junit4]   2> 2085320 INFO  (SocketProxy-Acceptor-34862) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=42388,localport=34862], receiveBufferSize:531000
   [junit4]   2> 2085328 INFO  (SocketProxy-Acceptor-34862) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=33061,localport=37486], receiveBufferSize=530904
   [junit4]   2> 2085365 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2085365 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2085382 INFO  (zkCallback-2073-thread-1-processing-n:127.0.0.1:34062_) [n:127.0.0.1:34062_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2085382 INFO  (zkCallback-2079-thread-1-processing-n:127.0.0.1:34862_) [n:127.0.0.1:34862_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2086488 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2086818 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.s.IndexSchema [collection1_shard1_replica_t43] Schema name=test
   [junit4]   2> 2087711 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2088204 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t43' using configuration from collection collection1, trusted=true
   [junit4]   2> 2088205 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_t43' (registry 'solr.core.collection1.shard1.replica_t43') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed
   [junit4]   2> 2088237 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2088238 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.SolrCore [[collection1_shard1_replica_t43] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/shard-2-001/cores/collection1_shard1_replica_t43], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/shard-2-001/cores/collection1_shard1_replica_t43/data/]
   [junit4]   2> 2088240 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8508779157034069]
   [junit4]   2> 2088260 WARN  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2088769 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2088769 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2088770 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2088770 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2088771 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=14.3369140625, floorSegmentMB=1.78515625, forceMergeDeletesPctAllowed=20.927308559770495, segmentsPerTier=49.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2088772 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.s.SolrIndexSearcher Opening [Searcher@4a59e38c[collection1_shard1_replica_t43] main]
   [junit4]   2> 2088785 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2088786 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2088786 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2088786 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578808189134045184
   [junit4]   2> 2088789 INFO  (searcherExecutor-2460-thread-1-processing-n:127.0.0.1:34862_ x:collection1_shard1_replica_t43 s:shard1 c:collection1) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.SolrCore [collection1_shard1_replica_t43] Registered new searcher Searcher@4a59e38c[collection1_shard1_replica_t43] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2088823 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2088823 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2088823 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:34862/collection1_shard1_replica_t43/
   [junit4]   2> 2088823 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2088823 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.SyncStrategy http://127.0.0.1:34862/collection1_shard1_replica_t43/ has no replicas
   [junit4]   2> 2088823 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2088824 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.ZkController collection1_shard1_replica_t43 stopping background replication from leader
   [junit4]   2> 2088825 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:34862/collection1_shard1_replica_t43/ shard1
   [junit4]   2> 2088843 INFO  (zkCallback-2079-thread-1-processing-n:127.0.0.1:34862_) [n:127.0.0.1:34862_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2088843 INFO  (zkCallback-2073-thread-1-processing-n:127.0.0.1:34062_) [n:127.0.0.1:34062_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2088965 INFO  (zkCallback-2073-thread-1-processing-n:127.0.0.1:34062_) [n:127.0.0.1:34062_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2088965 INFO  (zkCallback-2079-thread-1-processing-n:127.0.0.1:34862_) [n:127.0.0.1:34862_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2088996 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2088997 INFO  (qtp1633032030-8250) [n:127.0.0.1:34862_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=3632
   [junit4]   2> 2089000 INFO  (qtp163246312-8166) [n:127.0.0.1:40260_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:34862_&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2} status=0 QTime=3740
   [junit4]   2> 2089107 INFO  (zkCallback-2073-thread-1-processing-n:127.0.0.1:34062_) [n:127.0.0.1:34062_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2089122 INFO  (zkCallback-2079-thread-1-processing-n:127.0.0.1:34862_) [n:127.0.0.1:34862_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2089326 INFO  (OverseerCollectionConfigSetProcessor-98675510484533252-127.0.0.1:40260_-n_0000000000) [n:127.0.0.1:40260_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2090157 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/shard-3-001 of type TLOG
   [junit4]   2> 2090157 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2090337 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7f0b80bf{/,null,AVAILABLE}
   [junit4]   2> 2090337 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@48fa1cbb{HTTP/1.1,[http/1.1]}{127.0.0.1:33432}
   [junit4]   2> 2090338 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.e.j.s.Server Started @2104281ms
   [junit4]   2> 2090338 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/tempDir-001/jetty3, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=39486, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/shard-3-001/cores}
   [junit4]   2> 2090338 ERROR (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2090358 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2090358 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2090358 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 2090358 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-17T17:20:22.081Z
   [junit4]   2> 2090451 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2090451 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/shard-3-001/solr.xml
   [junit4]   2> 2090467 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2090525 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40266/solr
   [junit4]   2> 2090711 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:39486_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2090712 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:39486_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2090731 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:39486_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39486_
   [junit4]   2> 2090732 INFO  (zkCallback-2079-thread-1-processing-n:127.0.0.1:34862_) [n:127.0.0.1:34862_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2090784 INFO  (zkCallback-2061-thread-1-processing-n:127.0.0.1:40260_) [n:127.0.0.1:40260_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2090792 INFO  (zkCallback-2073-thread-1-processing-n:127.0.0.1:34062_) [n:127.0.0.1:34062_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2090792 INFO  (zkCallback-2068-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2090906 INFO  (zkCallback-2085-thread-1-processing-n:127.0.0.1:39486_) [n:127.0.0.1:39486_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2091466 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:39486_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed
   [junit4]   2> 2091592 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:39486_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed
   [junit4]   2> 2091592 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:39486_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2c77ed
   [junit4]   2> 2091626 INFO  (TEST-HttpPartitionTest.test-seed#[82F6963EC18A4E82]) [n:127.0.0.1:39486_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_82F6963EC18A4E82-001/shard-3-001/cores
   [junit4]   2> 2092178 INFO  (SocketProxy-Acceptor-39486) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=39112,localport=39486], receiveBufferSize:531000
   [junit4]   2> 2092198 INFO  (SocketProxy-Acceptor-39486) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=33432,localport=42024], receiveBufferSize=530904
   [junit4]   2> 2092273 INFO  (qtp1361356340-8283) [n:127.0.0.1:39486_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:39486_&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2092304 INFO  (OverseerThreadFactory-2435-thread-5-processing-n:127.0.0.1:40260_) [n:127.0.0.1:40260_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:39486_ for creating new replica
   [junit4]   2> 2092308 INFO  (SocketProxy-Acceptor-39486) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=39118,localport=39486], receiveBufferSize:531000
   [junit4]   2> 2092397 INFO  (SocketProxy-Acceptor-39486) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=33432,localport=42030], receiveBufferSize=530904
   [junit4]   2> 2092436 INFO  (qtp1361356340-8281) [n:127.0.0.1:39486_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t45&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2092437 INFO  (qtp1361356340-8281) [n:127.0.0.1:39486_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2092489 INFO  (zkCallback-2073-thread-1-processing-n:127.0.0.1:34062_) [n:127.0.0.1:34062_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2092489 INFO  (zkCallback-2079-thread-1-processing-n:127.0.0.1:34862_) [n:127.0.0.1:34862_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2092489 INFO  (zkCallback-2085-thread-1-processing-n:127.0.0.1:39486_) [n:127.0.0.1:39486_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2093594 INFO  (qtp1361356340-8281) [n:127.0.0.1:39486_ c:collection1 s:shard2  x:collection1_shard2_replica_t45] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2093722 INFO  (qtp1361356340-8281) [n:127.0.0.1:39486_ c:collection1 s:shard2  x:collection1_shard2_replica_t45] o.a.s.s.IndexSchema [collection1_shard2_replica_t45] Schema name=test
   [junit4]   2> 2094869 IN

[...truncated too long message...]

lection1_shard1_replica_t1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1.shard1.replica_t1, tag=1700535104
   [junit4]   2> 5426587 INFO  (recoveryExecutor-3281-thread-2-processing-n:127.0.0.1:37233_solr x:collection1_shard1_replica_t1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37233_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@73bfecca: rootName = solr_37233, domain = solr.core.collection1.shard1.replica_t1, service url = null, agent id = null] for registry solr.core.collection1.shard1.replica_t1 / com.codahale.metrics.MetricRegistry@7263091e
   [junit4]   2> 5426660 INFO  (recoveryExecutor-3281-thread-2-processing-n:127.0.0.1:37233_solr x:collection1_shard1_replica_t1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37233_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_t1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=1700535104
   [junit4]   2> 5426660 WARN  (recoveryExecutor-3281-thread-2-processing-n:127.0.0.1:37233_solr x:collection1_shard1_replica_t1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37233_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_t1] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection1_shard1_replica_t1] coreNodeName=[core_node2]
   [junit4]   2> 5426714 WARN  (updateExecutor-3280-thread-1-processing-n:127.0.0.1:37233_solr x:collection1_shard1_replica_t1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37233_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_t1] o.a.s.u.DefaultSolrCoreState Skipping recovery because Solr is shutdown
   [junit4]   2> 5426715 INFO  (jetty-closer-3247-thread-1) [    ] o.a.s.c.Overseer Overseer (id=98675718419578897-127.0.0.1:37233_solr-n_0000000004) closing
   [junit4]   2> 5426715 INFO  (OverseerStateUpdate-98675718419578897-127.0.0.1:37233_solr-n_0000000004) [n:127.0.0.1:37233_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37233_solr
   [junit4]   2> 5426862 WARN  (zkCallback-3289-thread-2-processing-n:127.0.0.1:34554_solr) [n:127.0.0.1:34554_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 5426862 INFO  (jetty-closer-3247-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4704b645{/solr,null,UNAVAILABLE}
   [junit4]   2> 5428235 WARN  (zkCallback-3283-thread-3-processing-n:127.0.0.1:37233_solr) [n:127.0.0.1:37233_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 5428246 INFO  (jetty-closer-3247-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4483896e{/solr,null,UNAVAILABLE}
   [junit4]   2> 5428254 ERROR (SUITE-TestCloudRecovery-seed#[82F6963EC18A4E82]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 5428254 INFO  (SUITE-TestCloudRecovery-seed#[82F6963EC18A4E82]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:44911 44911
   [junit4]   2> 5428388 INFO  (Thread-7177) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:44911 44911
   [junit4]   2> 5428391 WARN  (Thread-7177) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 7 /solr/aliases.json
   [junit4]   2> 7 /solr/clusterprops.json
   [junit4]   2> 6 /solr/security.json
   [junit4]   2> 6 /solr/configs/config
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 30 /solr/collections/collection1/state.json
   [junit4]   2> 7 /solr/clusterstate.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 7 /solr/collections
   [junit4]   2> 5 /solr/overseer/queue
   [junit4]   2> 5 /solr/overseer/collection-queue-work
   [junit4]   2> 3 /solr/live_nodes
   [junit4]   2> 3 /solr/overseer/queue-work
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.TestCloudRecovery_82F6963EC18A4E82-001
   [junit4]   2> Sep 17, 2017 6:16:00 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=false): {}, locale=mt-MT, timezone=Europe/San_Marino
   [junit4]   2> NOTE: Linux 4.4.0-83-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=109092096,total=522715136
   [junit4]   2> NOTE: All tests run in this JVM: [TestSystemCollAutoCreate, TestCloudSchemaless, TestConfigSets, TestLocalFSCloudBackupRestore, TestTolerantSearch, DistributedFacetExistsSmallTest, TestApiFramework, HdfsRecoveryZkTest, BadIndexSchemaTest, MergeStrategyTest, CopyFieldTest, TestClusterProperties, RequiredFieldsTest, TestJavabinTupleStreamParser, AutoScalingHandlerTest, AnalysisErrorHandlingTest, XsltUpdateRequestHandlerTest, TestInPlaceUpdatesStandalone, SuggesterFSTTest, TestJsonFacets, UUIDUpdateProcessorFallbackTest, TestCloudInspectUtil, UnloadDistributedZkTest, PathHierarchyTokenizerFactoryTest, TestTrie, TestExceedMaxTermLength, TestDFRSimilarityFactory, TestBulkSchemaAPI, TestSearcherReuse, OverseerRolesTest, TestUseDocValuesAsStored, ReturnFieldsTest, ShufflingReplicaListTransformerTest, CheckHdfsIndexTest, TestDynamicLoading, ExplicitHLLTest, SignatureUpdateProcessorFactoryTest, TestMissingGroups, TestFoldingMultitermQuery, BooleanFieldTest, CloneFieldUpdateProcessorFactoryTest, SimpleCollectionCreateDeleteTest, HdfsBasicDistributedZkTest, HdfsWriteToMultipleCollectionsTest, FacetPivotSmallTest, TestMacroExpander, TermsComponentTest, TestStressReorder, HLLUtilTest, MoveReplicaHDFSTest, AssignTest, RegexBoostProcessorTest, OverseerTest, CdcrRequestHandlerTest, TestSubQueryTransformer, TestInitQParser, TestCloudNestedDocsSort, DirectUpdateHandlerTest, PeerSyncWithIndexFingerprintCachingTest, RequestLoggingTest, TestSQLHandlerNonCloud, PolyFieldTest, TestPullReplicaErrorHandling, ParsingFieldUpdateProcessorsTest, BlockJoinFacetRandomTest, TestConfigReload, TestGroupingSearch, TestSolrXml, TimeZoneUtilsTest, SolrMetricsIntegrationTest, TestSortByMinMaxFunction, TestPKIAuthenticationPlugin, TestSolr4Spatial2, TestQuerySenderListener, URLClassifyProcessorTest, TestConfigSetsAPI, ZkStateReaderTest, TestSuggestSpellingConverter, TestDelegationWithHadoopAuth, SmileWriterTest, TestExportWriter, CircularListTest, TriLevelCompositeIdRoutingTest, DistributedVersionInfoTest, TestTolerantUpdateProcessorCloud, TestRequestStatusCollectionAPI, TestBlendedInfixSuggestions, TestCloudPseudoReturnFields, TestAddFieldRealTimeGet, TestSort, LeaderInitiatedRecoveryOnShardRestartTest, TestAnalyzedSuggestions, TestHighFrequencyDictionaryFactory, TestShortCircuitedRequests, RollingRestartTest, TestRebalanceLeaders, TestShardHandlerFactory, DateMathParserTest, CoreSorterTest, TestJmxIntegration, SuggestComponentTest, QueryEqualityTest, HdfsCollectionsAPIDistributedZkTest, ClusterStateTest, ShowFileRequestHandlerTest, TestTestInjection, DirectUpdateHandlerOptimizeTest, TestExactStatsCache, TestInitParams, TestRecovery, SpatialRPTFieldTypeTest, TestFastWriter, AutoCommitTest, DistributedQueryComponentCustomSortTest, SolrTestCaseJ4Test, TestLeaderElectionWithEmptyReplica, TestUtils, TestZkAclsWithHadoopAuth, SpellPossibilityIteratorTest, ConjunctionSolrSpellCheckerTest, BasicDistributedZk2Test, SyncSliceTest, OpenCloseCoreStressTest, LeaderElectionIntegrationTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, LeaderElectionTest, TestZkChroot, ZkCLITest, TestFaceting, TestHashPartitioner, TestReload, TestCoreContainer, TestSolr4Spatial, QueryElevationComponentTest, PeerSyncTest, TestFunctionQuery, TestLazyCores, DistributedQueryElevationComponentTest, SolrIndexSplitterTest, SimplePostToolTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SpellCheckCollatorTest, SuggesterTest, SuggesterWFSTTest, TestCSVLoader, SolrCoreCheckLockOnStartupTest, TestPseudoReturnFields, FieldMutatingUpdateProcessorTest, TestAtomicUpdateErrorCases, StatelessScriptUpdateProcessorFactoryTest, SortByFunctionTest, TestRemoteStreaming, DocValuesMultiTest, SolrInfoBeanTest, CacheHeaderTest, TestSurroundQueryParser, LukeRequestHandlerTest, DocumentAnalysisRequestHandlerTest, XmlUpdateRequestHandlerTest, TermVectorComponentTest, FieldAnalysisRequestHandlerTest, IndexSchemaRuntimeFieldTest, LoggingHandlerTest, QueryParsingTest, BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, TestLFUCache, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestEmbeddedSolrServerConstructors, TestEmbeddedSolrServerSchemaAPI, ConnectionReuseTest, AsyncCallRequestStatusResponseTest, CdcrBootstrapTest, CdcrReplicationHandlerTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, CollectionTooManyReplicasTest, ConcurrentDeleteAndCreateCollectionTest, ConfigSetsAPITest, ConnectionManagerTest, DeleteLastCustomShardedReplicaTest, DeleteNodeTest, DeleteShardTest, DistribCursorPagingTest, LeaderInitiatedRecoveryOnCommitTest, MoveReplicaHDFSUlogDirTest, NodeMutatorTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, OverseerStatusTest, PeerSyncReplicationTest, SolrXmlInZkTest, TestAuthenticationFramework, TestCloudRecovery]
   [junit4] Completed [545/732 (3!)] on J2 in 188.25s, 2 tests, 1 failure <<< FAILURES!

[...truncated 46789 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-master - Build # 2102 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/2102/

7 tests failed.
FAILED:  org.apache.solr.cloud.AssignTest.testIdIsUnique

Error Message:
org.apache.solr.common.SolrException: Error inc and get counter from Zookeeper for collection:c4

Stack Trace:
java.util.concurrent.ExecutionException: org.apache.solr.common.SolrException: Error inc and get counter from Zookeeper for collection:c4
        at __randomizedtesting.SeedInfo.seed([356A5B4AD3E160EC:A7D7F762443CE01A]:0)
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.solr.cloud.AssignTest.testIdIsUnique(AssignTest.java:114)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.common.SolrException: Error inc and get counter from Zookeeper for collection:c4
        at org.apache.solr.cloud.Assign.incAndGetId(Assign.java:107)
        at org.apache.solr.cloud.AssignTest.lambda$testIdIsUnique$2(AssignTest.java:106)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/c4/counter
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1327)
        at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:374)
        at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:371)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:371)
        at org.apache.solr.cloud.Assign.incAndGetId(Assign.java:103)
        ... 7 more


FAILED:  org.apache.solr.cloud.UnloadDistributedZkTest.test

Error Message:
Could not load collection from ZK: unloadcollection

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: unloadcollection
        at __randomizedtesting.SeedInfo.seed([356A5B4AD3E160EC:BD3E64907D1D0D14]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1115)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:128)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:154)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:140)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:135)
        at org.apache.solr.cloud.UnloadDistributedZkTest.testCoreUnloadAndLeaders(UnloadDistributedZkTest.java:221)
        at org.apache.solr.cloud.UnloadDistributedZkTest.test(UnloadDistributedZkTest.java:68)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/unloadcollection/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1127)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1113)
        ... 48 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest

Error Message:
ObjectTracker found 2 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:478)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:928)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1002)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1002)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 2 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:478)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:928)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1002)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1002)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)


        at __randomizedtesting.SeedInfo.seed([356A5B4AD3E160EC]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:294)
        at sun.reflect.GeneratedMethodAccessor131.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:897)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig

Error Message:
The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.

Stack Trace:
java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
        at __randomizedtesting.SeedInfo.seed([356A5B4AD3E160EC:C2C592612E688AC7]:0)
        at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:311)
        at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:287)
        at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:227)
        at org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig(HdfsDirectoryFactoryTest.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
        at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:71)
        at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:309)
        ... 42 more


FAILED:  org.apache.solr.core.TestLazyCores.testRace

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([356A5B4AD3E160EC]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([356A5B4AD3E160EC]:0)


FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
KeeperErrorCode = Session expired for /clusterstate.json

Stack Trace:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
        at __randomizedtesting.SeedInfo.seed([356A5B4AD3E160EC:BD3E64907D1D0D14]:0)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.refreshLegacyClusterState(ZkStateReader.java:542)
        at org.apache.solr.common.cloud.ZkStateReader.forceUpdateCollection(ZkStateReader.java:310)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.updateMappingsFromZk(AbstractFullDistribZkTestBase.java:674)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.updateMappingsFromZk(AbstractFullDistribZkTestBase.java:669)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:464)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:334)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11375 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> 1385440 INFO  (SUITE-HttpPartitionTest-seed#[356A5B4AD3E160EC]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/init-core-data-001
   [junit4]   2> 1385441 WARN  (SUITE-HttpPartitionTest-seed#[356A5B4AD3E160EC]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=26 numCloses=26
   [junit4]   2> 1385441 INFO  (SUITE-HttpPartitionTest-seed#[356A5B4AD3E160EC]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1385456 INFO  (SUITE-HttpPartitionTest-seed#[356A5B4AD3E160EC]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 1385456 INFO  (SUITE-HttpPartitionTest-seed#[356A5B4AD3E160EC]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1385472 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1385472 INFO  (Thread-326) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1385472 INFO  (Thread-326) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1385612 ERROR (Thread-326) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1385612 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.ZkTestServer start zk server on port:42264
   [junit4]   2> 1385784 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1385792 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1385852 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1385884 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1385885 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1385886 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1385886 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1385887 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1385892 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1385893 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1385904 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1385948 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use TLOG replicas unless explicitly asked otherwise
   [junit4]   2> 1386794 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1386812 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6d55ccff{/,null,AVAILABLE}
   [junit4]   2> 1386814 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2951d95c{HTTP/1.1,[http/1.1]}{127.0.0.1:36222}
   [junit4]   2> 1386814 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.e.j.s.Server Started @1406237ms
   [junit4]   2> 1386814 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=56045, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/control-001/cores}
   [junit4]   2> 1386814 ERROR (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1386814 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1386814 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1386814 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1386814 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-18T22:35:42.002Z
   [junit4]   2> 1386908 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1386908 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/control-001/solr.xml
   [junit4]   2> 1386948 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@77fcc942, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1386953 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42264/solr
   [junit4]   2> 1387566 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:56045_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1387566 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:56045_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:56045_
   [junit4]   2> 1387567 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:56045_    ] o.a.s.c.Overseer Overseer (id=98682414082818052-127.0.0.1:56045_-n_0000000000) starting
   [junit4]   2> 1387968 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:56045_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56045_
   [junit4]   2> 1387969 INFO  (zkCallback-1078-thread-1-processing-n:127.0.0.1:56045_) [n:127.0.0.1:56045_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1389385 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:56045_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77fcc942
   [junit4]   2> 1389467 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:56045_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77fcc942
   [junit4]   2> 1389472 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:56045_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77fcc942
   [junit4]   2> 1389474 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:56045_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/control-001/cores
   [junit4]   2> 1390629 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1390629 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42264/solr ready
   [junit4]   2> 1391368 INFO  (SocketProxy-Acceptor-56045) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=43515,localport=56045], receiveBufferSize:531000
   [junit4]   2> 1391381 INFO  (SocketProxy-Acceptor-56045) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=36222,localport=38342], receiveBufferSize=530904
   [junit4]   2> 1391453 INFO  (qtp111138694-3294) [n:127.0.0.1:56045_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:56045_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1391512 INFO  (OverseerThreadFactory-770-thread-1-processing-n:127.0.0.1:56045_) [n:127.0.0.1:56045_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 1391898 INFO  (SocketProxy-Acceptor-56045) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=43517,localport=56045], receiveBufferSize:531000
   [junit4]   2> 1391906 INFO  (SocketProxy-Acceptor-56045) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=36222,localport=38344], receiveBufferSize=530904
   [junit4]   2> 1391957 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1391958 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1392086 INFO  (zkCallback-1078-thread-1-processing-n:127.0.0.1:56045_) [n:127.0.0.1:56045_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1393380 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1393609 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 1395124 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1395301 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 1395302 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77fcc942
   [junit4]   2> 1395302 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1395302 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 1395348 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=23, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1395407 WARN  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1396645 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1396645 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1396646 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1396646 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1398397 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.38613845082186804]
   [junit4]   2> 1398404 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@d5da1e2[control_collection_shard1_replica_n1] main]
   [junit4]   2> 1398406 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1398406 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1398407 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1398409 INFO  (searcherExecutor-773-thread-1-processing-n:127.0.0.1:56045_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@d5da1e2[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1398409 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578918638882127872
   [junit4]   2> 1398417 INFO  (zkCallback-1078-thread-1-processing-n:127.0.0.1:56045_) [n:127.0.0.1:56045_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1398608 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1398608 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1398608 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:56045/control_collection_shard1_replica_n1/
   [junit4]   2> 1398608 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1398608 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:56045/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 1398608 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1398632 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:56045/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 1398786 INFO  (zkCallback-1078-thread-1-processing-n:127.0.0.1:56045_) [n:127.0.0.1:56045_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1398788 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1398960 INFO  (zkCallback-1078-thread-2-processing-n:127.0.0.1:56045_) [n:127.0.0.1:56045_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1399036 INFO  (qtp111138694-3295) [n:127.0.0.1:56045_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=7079
   [junit4]   2> 1399058 INFO  (qtp111138694-3294) [n:127.0.0.1:56045_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1399058 INFO  (qtp111138694-3294) [n:127.0.0.1:56045_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:56045_&wt=javabin&version=2} status=0 QTime=7605
   [junit4]   2> 1399069 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15e9721fcfc0005, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1399191 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1399191 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42264/solr ready
   [junit4]   2> 1399191 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1399196 INFO  (SocketProxy-Acceptor-56045) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=43525,localport=56045], receiveBufferSize:531000
   [junit4]   2> 1399208 INFO  (SocketProxy-Acceptor-56045) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=36222,localport=38352], receiveBufferSize=530904
   [junit4]   2> 1399229 INFO  (qtp111138694-3301) [n:127.0.0.1:56045_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1399231 INFO  (OverseerCollectionConfigSetProcessor-98682414082818052-127.0.0.1:56045_-n_0000000000) [n:127.0.0.1:56045_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1399336 INFO  (OverseerThreadFactory-770-thread-2-processing-n:127.0.0.1:56045_) [n:127.0.0.1:56045_    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 1399337 WARN  (OverseerThreadFactory-770-thread-2-processing-n:127.0.0.1:56045_) [n:127.0.0.1:56045_    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 1399569 INFO  (qtp111138694-3301) [n:127.0.0.1:56045_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1399569 INFO  (qtp111138694-3301) [n:127.0.0.1:56045_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=340
   [junit4]   2> 1400542 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/shard-1-001 of type TLOG
   [junit4]   2> 1400549 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1400553 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@31f437ed{/,null,AVAILABLE}
   [junit4]   2> 1400553 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@42022943{HTTP/1.1,[http/1.1]}{127.0.0.1:52615}
   [junit4]   2> 1400553 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.e.j.s.Server Started @1419977ms
   [junit4]   2> 1400553 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/tempDir-001/jetty1, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=47747, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/shard-1-001/cores}
   [junit4]   2> 1400554 ERROR (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1400554 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1400554 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1400554 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1400554 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-18T22:35:55.742Z
   [junit4]   2> 1400640 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1400640 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/shard-1-001/solr.xml
   [junit4]   2> 1400680 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@77fcc942, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1400682 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42264/solr
   [junit4]   2> 1401481 INFO  (OverseerCollectionConfigSetProcessor-98682414082818052-127.0.0.1:56045_-n_0000000000) [n:127.0.0.1:56045_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1402193 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:47747_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1402204 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:47747_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1402288 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:47747_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47747_
   [junit4]   2> 1402289 INFO  (zkCallback-1085-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1402289 INFO  (zkCallback-1078-thread-2-processing-n:127.0.0.1:56045_) [n:127.0.0.1:56045_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1403029 INFO  (zkCallback-1090-thread-1-processing-n:127.0.0.1:47747_) [n:127.0.0.1:47747_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1405545 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:47747_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77fcc942
   [junit4]   2> 1405618 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:47747_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77fcc942
   [junit4]   2> 1405618 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:47747_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77fcc942
   [junit4]   2> 1405648 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:47747_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/shard-1-001/cores
   [junit4]   2> 1405987 INFO  (SocketProxy-Acceptor-47747) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=56961,localport=47747], receiveBufferSize:531000
   [junit4]   2> 1406097 INFO  (SocketProxy-Acceptor-47747) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=52615,localport=60245], receiveBufferSize=530904
   [junit4]   2> 1406113 INFO  (qtp2023569276-3347) [n:127.0.0.1:47747_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:47747_&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1406144 INFO  (OverseerThreadFactory-770-thread-3-processing-n:127.0.0.1:56045_) [n:127.0.0.1:56045_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:47747_ for creating new replica
   [junit4]   2> 1406169 INFO  (SocketProxy-Acceptor-47747) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=56963,localport=47747], receiveBufferSize:531000
   [junit4]   2> 1406456 INFO  (SocketProxy-Acceptor-47747) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=52615,localport=60247], receiveBufferSize=530904
   [junit4]   2> 1406497 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 1406500 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1408351 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1408450 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.IndexSchema [collection1_shard2_replica_t41] Schema name=test
   [junit4]   2> 1409985 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1410041 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_t41' using configuration from collection collection1, trusted=true
   [junit4]   2> 1410042 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_t41' (registry 'solr.core.collection1.shard2.replica_t41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77fcc942
   [junit4]   2> 1410042 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1410042 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SolrCore [[collection1_shard2_replica_t41] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/shard-1-001/cores/collection1_shard2_replica_t41], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/shard-1-001/cores/collection1_shard2_replica_t41/data/]
   [junit4]   2> 1410138 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=23, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1410182 WARN  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1410644 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1410644 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1410645 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1410645 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1410669 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.38613845082186804]
   [junit4]   2> 1410720 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.SolrIndexSearcher Opening [Searcher@2bb9a0db[collection1_shard2_replica_t41] main]
   [junit4]   2> 1410725 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1410725 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1410736 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1410736 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578918651807924224
   [junit4]   2> 1410838 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1410838 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1410838 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:47747/collection1_shard2_replica_t41/
   [junit4]   2> 1410838 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1410838 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SyncStrategy http://127.0.0.1:47747/collection1_shard2_replica_t41/ has no replicas
   [junit4]   2> 1410838 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1410838 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ZkController collection1_shard2_replica_t41 stopping background replication from leader
   [junit4]   2> 1410853 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:47747/collection1_shard2_replica_t41/ shard2
   [junit4]   2> 1411185 INFO  (searcherExecutor-784-thread-1-processing-n:127.0.0.1:47747_ x:collection1_shard2_replica_t41 s:shard2 c:collection1) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SolrCore [collection1_shard2_replica_t41] Registered new searcher Searcher@2bb9a0db[collection1_shard2_replica_t41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1411544 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1411545 INFO  (qtp2023569276-3345) [n:127.0.0.1:47747_ c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=5048
   [junit4]   2> 1411605 INFO  (qtp2023569276-3347) [n:127.0.0.1:47747_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:47747_&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2} status=0 QTime=5492
   [junit4]   2> 1412867 INFO  (OverseerCollectionConfigSetProcessor-98682414082818052-127.0.0.1:56045_-n_0000000000) [n:127.0.0.1:56045_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1413268 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/shard-2-001 of type TLOG
   [junit4]   2> 1413270 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1413273 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@66db8e11{/,null,AVAILABLE}
   [junit4]   2> 1413273 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@29e97286{HTTP/1.1,[http/1.1]}{127.0.0.1:46091}
   [junit4]   2> 1413273 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.e.j.s.Server Started @1432697ms
   [junit4]   2> 1413273 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/tempDir-001/jetty2, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=46581, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/shard-2-001/cores}
   [junit4]   2> 1413274 ERROR (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1413284 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1413284 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1413284 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1413284 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-18T22:36:08.472Z
   [junit4]   2> 1413405 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1413405 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/shard-2-001/solr.xml
   [junit4]   2> 1413425 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@77fcc942, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1413430 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42264/solr
   [junit4]   2> 1413636 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:46581_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1413657 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:46581_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1413658 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:46581_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46581_
   [junit4]   2> 1413658 INFO  (zkCallback-1085-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1413760 INFO  (zkCallback-1096-thread-1-processing-n:127.0.0.1:46581_) [n:127.0.0.1:46581_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1413746 INFO  (zkCallback-1090-thread-1-processing-n:127.0.0.1:47747_) [n:127.0.0.1:47747_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1413692 INFO  (zkCallback-1078-thread-1-processing-n:127.0.0.1:56045_) [n:127.0.0.1:56045_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1414395 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:46581_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77fcc942
   [junit4]   2> 1414479 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:46581_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77fcc942
   [junit4]   2> 1414479 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:46581_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@77fcc942
   [junit4]   2> 1414488 INFO  (TEST-HttpPartitionTest.test-seed#[356A5B4AD3E160EC]) [n:127.0.0.1:46581_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_356A5B4AD3E160EC-001/shard-2-001/cores
   [junit4]   2> 1414949 INFO  (SocketProxy-Acceptor-46581) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=36565,localport=46581], receiveBufferSize:531000
   [junit4]   2> 1414952 INFO  (SocketProxy-Acceptor-46581) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=46091,localport=47901], receiveBufferSize=530904
   [junit4]   2> 1414963 INFO  (qtp241990181-3383) [n:127.0.0.1:46581_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:46581_&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1414966 INFO  (OverseerThreadFactory-770-thread-4-processing-n:127.0.0.1:56045_) [n:127.0.0.1:56045_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:46581_ for creating new replica
   [junit4]   2> 1414977 INFO  (SocketProxy-Acceptor-46581) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=36567,localport=46581], receiveBufferSize:531000
   [junit4]   2> 1414977 INFO  (SocketProxy-Acceptor-46581) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=46091,localport=47903], receiveBufferSize=530904
   [junit4]   2> 1415016 INFO  (qtp241990181-3381) [n:127.0.0.1:46581_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 1415017 INFO  (qtp241990181-3381) [n:127.0.0.1:46581_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1416063 INFO  (qtp241990181-3381) [n:127.0.0.1:46581_ c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1416202 INFO  (qtp241990181-3381) [n:127.0.0.1:46581_ c:collection1 s:shard1  x:c

[...truncated too long message...]

ros-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-interceptors-admin-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-interceptors-authn-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-interceptors-authz-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-interceptors-changelog-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/byte-buddy-1.6.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/commons-math3-3.6.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/hadoop-common-2.7.4-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/hadoop-hdfs-2.7.4-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/hadoop-minikdc-2.7.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/jetty-sslengine-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/mockito-core-2.6.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/test-lib/objenesis-2.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/build/analysis/icu/lucene-analyzers-icu-8.0.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/analysis-extras/lib/icu4j-59.1.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/usr/local/asfpackages/java/jdk1.8.0_144/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.3.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/temp/junit4-J1-20170918_221215_0345578861763487543957.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/temp/junit4-J1-20170918_221215_0347087530923509222697.suites -stdin
   [junit4] ERROR: JVM J1 ended with an exception: Forked process returned with error code: 1. Very likely a JVM crash.  See process stdout at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/temp/junit4-J1-20170918_221215_0346290365978161975361.sysout See process stderr at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/temp/junit4-J1-20170918_221215_0344304265800365562717.syserr
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1519)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4] at java.lang.Thread.run(Thread.java:748)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:826: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:770: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build.xml:267: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/common-build.xml:549: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/common-build.xml:1494: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/common-build.xml:1021: At least one slave process threw an exception, first: Forked process returned with error code: 1. Very likely a JVM crash.  See process stdout at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/temp/junit4-J1-20170918_221215_0346290365978161975361.sysout See process stderr at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/temp/junit4-J1-20170918_221215_0344304265800365562717.syserr

Total time: 390 minutes 44 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[Fast Archiver] No artifacts from Lucene-Solr-Tests-master #2101 to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-master - Build # 2103 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/2103/

14 tests failed.
FAILED:  org.apache.solr.cloud.CdcrBootstrapTest.testBootstrapWithSourceCluster

Error Message:
Captured an uncaught exception in thread: Thread[id=14845, name=Thread-3484, state=RUNNABLE, group=TGRP-CdcrBootstrapTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=14845, name=Thread-3484, state=RUNNABLE, group=TGRP-CdcrBootstrapTest]
        at __randomizedtesting.SeedInfo.seed([4A7ECA3F1D8FF724:93289BFB1EEBE46E]:0)
Caused by: java.lang.AssertionError: 1
        at __randomizedtesting.SeedInfo.seed([4A7ECA3F1D8FF724]:0)
        at org.apache.solr.core.CachingDirectoryFactory.close(CachingDirectoryFactory.java:192)
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1613)
        at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:863)
        at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1236)
        at org.apache.solr.handler.IndexFetcher.lambda$reloadCore$0(IndexFetcher.java:900)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrBootstrapTest

Error Message:
ObjectTracker found 5 object(s) that were not released!!! [MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:478)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:928)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1002)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:349)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:709)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1002)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:484)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:332)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:419)  at org.apache.solr.handler.CdcrRequestHandler$BootstrapCallable.call(CdcrRequestHandler.java:761)  at org.apache.solr.handler.CdcrRequestHandler$BootstrapCallable.call(CdcrRequestHandler.java:716)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1002)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1019)  at org.apache.solr.core.SolrCore.reload(SolrCore.java:636)  at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1235)  at org.apache.solr.handler.IndexFetcher.lambda$reloadCore$0(IndexFetcher.java:900)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 5 object(s) that were not released!!! [MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:478)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:928)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1002)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:349)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:709)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1002)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:484)
        at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:332)
        at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:419)
        at org.apache.solr.handler.CdcrRequestHandler$BootstrapCallable.call(CdcrRequestHandler.java:761)
        at org.apache.solr.handler.CdcrRequestHandler$BootstrapCallable.call(CdcrRequestHandler.java:716)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1002)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1019)
        at org.apache.solr.core.SolrCore.reload(SolrCore.java:636)
        at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1235)
        at org.apache.solr.handler.IndexFetcher.lambda$reloadCore$0(IndexFetcher.java:900)
        at java.lang.Thread.run(Thread.java:748)


        at __randomizedtesting.SeedInfo.seed([4A7ECA3F1D8FF724]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:294)
        at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:897)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.DistribCursorPagingTest.test

Error Message:
Could not load collection from ZK: collection1

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at __randomizedtesting.SeedInfo.seed([4A7ECA3F1D8FF724:C22AF5E5B3739ADC]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1115)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1227)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
        at org.apache.solr.cloud.DistribCursorPagingTest.assertFullWalkNoDups(DistribCursorPagingTest.java:718)
        at org.apache.solr.cloud.DistribCursorPagingTest.doRandomSortsOnLargeIndex(DistribCursorPagingTest.java:545)
        at org.apache.solr.cloud.DistribCursorPagingTest.test(DistribCursorPagingTest.java:93)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1127)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1113)
        ... 51 more


FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
Expected 2 of 3 replicas to be active but only found 1; [core_node5:{"core":"c8n_1x3_lf_shard1_replica_t2","base_url":"http://127.0.0.1:59784/_jym/r","node_name":"127.0.0.1:59784__jym%2Fr","state":"active","type":"TLOG","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/16)={   "pullReplicas":"0",   "replicationFactor":"1",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node3":{           "core":"c8n_1x3_lf_shard1_replica_t1",           "base_url":"http://127.0.0.1:58207/_jym/r",           "node_name":"127.0.0.1:58207__jym%2Fr",           "state":"down",           "type":"TLOG"},         "core_node5":{           "core":"c8n_1x3_lf_shard1_replica_t2",           "base_url":"http://127.0.0.1:59784/_jym/r",           "node_name":"127.0.0.1:59784__jym%2Fr",           "state":"active",           "type":"TLOG",           "leader":"true"},         "core_node6":{           "state":"down",           "base_url":"http://127.0.0.1:40669/_jym/r",           "core":"c8n_1x3_lf_shard1_replica_t4",           "node_name":"127.0.0.1:40669__jym%2Fr",           "type":"TLOG"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"0",   "tlogReplicas":"3"}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node5:{"core":"c8n_1x3_lf_shard1_replica_t2","base_url":"http://127.0.0.1:59784/_jym/r","node_name":"127.0.0.1:59784__jym%2Fr","state":"active","type":"TLOG","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/16)={
  "pullReplicas":"0",
  "replicationFactor":"1",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "core":"c8n_1x3_lf_shard1_replica_t1",
          "base_url":"http://127.0.0.1:58207/_jym/r",
          "node_name":"127.0.0.1:58207__jym%2Fr",
          "state":"down",
          "type":"TLOG"},
        "core_node5":{
          "core":"c8n_1x3_lf_shard1_replica_t2",
          "base_url":"http://127.0.0.1:59784/_jym/r",
          "node_name":"127.0.0.1:59784__jym%2Fr",
          "state":"active",
          "type":"TLOG",
          "leader":"true"},
        "core_node6":{
          "state":"down",
          "base_url":"http://127.0.0.1:40669/_jym/r",
          "core":"c8n_1x3_lf_shard1_replica_t4",
          "node_name":"127.0.0.1:40669__jym%2Fr",
          "type":"TLOG"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"0",
  "tlogReplicas":"3"}
        at __randomizedtesting.SeedInfo.seed([4A7ECA3F1D8FF724:C22AF5E5B3739ADC]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:169)
        at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:56)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete

Error Message:
Error from server at http://127.0.0.1:44418/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 </title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason: <pre>    Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/> </body> </html>

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://127.0.0.1:44418/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>

        at __randomizedtesting.SeedInfo.seed([4A7ECA3F1D8FF724:E984649A9A671D81]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:539)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:993)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
        at org.apache.solr.cloud.TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete(TestCollectionsAPIViaSolrCloudCluster.java:167)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:44418/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>

        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:591)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:516)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more


FAILED:  org.apache.solr.cloud.TestCryptoKeys.test

Error Message:
{   "responseHeader":{     "status":500,     "QTime":34563},   "errorMessages":["4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:38648/jw_/collection1_shard1_replica_n21/, https://127.0.0.1:38012/jw_/collection1_shard1_replica_n23/, https://127.0.0.1:48205/jw_/collection1_shard1_replica_n25/, https://127.0.0.1:37985/jw_/collection1_shard1_replica_n27/]\n"],   "WARNING":"This response format is experimental.  It is likely to change in the future.",   "error":{     "metadata":[       "error-class","org.apache.solr.common.SolrException",       "root-error-class","org.apache.solr.common.SolrException"],     "msg":"4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:38648/jw_/collection1_shard1_replica_n21/, https://127.0.0.1:38012/jw_/collection1_shard1_replica_n23/, https://127.0.0.1:48205/jw_/collection1_shard1_replica_n25/, https://127.0.0.1:37985/jw_/collection1_shard1_replica_n27/]",     "trace":"org.apache.solr.common.SolrException: 4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:38648/jw_/collection1_shard1_replica_n21/, https://127.0.0.1:38012/jw_/collection1_shard1_replica_n23/, https://127.0.0.1:48205/jw_/collection1_shard1_replica_n25/, https://127.0.0.1:37985/jw_/collection1_shard1_replica_n27/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:772)\n\tat org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:102)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:503)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:346)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:159)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:137)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2483)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:748)\n",     "code":500}}

Stack Trace:
java.lang.AssertionError: {
  "responseHeader":{
    "status":500,
    "QTime":34563},
  "errorMessages":["4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:38648/jw_/collection1_shard1_replica_n21/, https://127.0.0.1:38012/jw_/collection1_shard1_replica_n23/, https://127.0.0.1:48205/jw_/collection1_shard1_replica_n25/, https://127.0.0.1:37985/jw_/collection1_shard1_replica_n27/]\n"],
  "WARNING":"This response format is experimental.  It is likely to change in the future.",
  "error":{
    "metadata":[
      "error-class","org.apache.solr.common.SolrException",
      "root-error-class","org.apache.solr.common.SolrException"],
    "msg":"4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:38648/jw_/collection1_shard1_replica_n21/, https://127.0.0.1:38012/jw_/collection1_shard1_replica_n23/, https://127.0.0.1:48205/jw_/collection1_shard1_replica_n25/, https://127.0.0.1:37985/jw_/collection1_shard1_replica_n27/]",
    "trace":"org.apache.solr.common.SolrException: 4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:38648/jw_/collection1_shard1_replica_n21/, https://127.0.0.1:38012/jw_/collection1_shard1_replica_n23/, https://127.0.0.1:48205/jw_/collection1_shard1_replica_n25/, https://127.0.0.1:37985/jw_/collection1_shard1_replica_n27/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:772)\n\tat org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:102)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:503)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:346)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:159)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:137)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2483)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:748)\n",
    "code":500}}

        at __randomizedtesting.SeedInfo.seed([4A7ECA3F1D8FF724:C22AF5E5B3739ADC]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.core.TestSolrConfigHandler.runConfigCommand(TestSolrConfigHandler.java:185)
        at org.apache.solr.cloud.TestCryptoKeys.test(TestCryptoKeys.java:127)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestRequestStatusCollectionAPI.test

Error Message:
Could not load collection from ZK: collection1

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at __randomizedtesting.SeedInfo.seed([4A7ECA3F1D8FF724:C22AF5E5B3739ADC]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1115)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:128)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.getTotalReplicas(AbstractFullDistribZkTestBase.java:488)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:441)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:334)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1127)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1113)
        ... 38 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestTlogReplica

Error Message:
ObjectTracker found 5 object(s) that were not released!!! [MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1002)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1019)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1002)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:478)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:928)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1002)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:484)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:332)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:419)  at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$12(ReplicationHandler.java:1183)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:349)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:709)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1002)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 5 object(s) that were not released!!! [MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1002)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleas

[...truncated too long message...]

eFromDescriptor(CoreContainer.java:1018)
   [junit4]   2> at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1444)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:267)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:480)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
   [junit4]   2> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:988)
   [junit4]   2> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
   [junit4]   2> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1002)
   [junit4]   2> ... 29 more
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
   [junit4]   2> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2075)
   [junit4]   2> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2195)
   [junit4]   2> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1071)
   [junit4]   2> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:960)
   [junit4]   2> ... 31 more
   [junit4]   2> Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrClient_67CE578726E381E8-001/instance-0-001/collection1/data/index/write.lock
   [junit4]   2> at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:139)
   [junit4]   2> at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41)
   [junit4]   2> at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45)
   [junit4]   2> at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:104)
   [junit4]   2> at org.apache.lucene.store.MockDirectoryWrapper.obtainLock(MockDirectoryWrapper.java:1049)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:947)
   [junit4]   2> at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:119)
   [junit4]   2> at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:94)
   [junit4]   2> at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:257)
   [junit4]   2> at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:131)
   [junit4]   2> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2036)
   [junit4]   2> ... 34 more
   [junit4]   2>
   [junit4]   2> 141218 INFO  (qtp937206892-289) [    ] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&distrib=false&sort=_docid_+asc&rows=0&wt=javabin&version=2} hits=10 status=0 QTime=0
   [junit4]   2> 141901 INFO  (qtp937206892-290) [    ] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=10 status=0 QTime=0
   [junit4]   2> 141970 INFO  (TEST-TestLBHttpSolrClient.testTwoServers-seed#[67CE578726E381E8]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@1d9996e9{HTTP/1.1,[http/1.1]}{127.0.0.1:38211}
   [junit4]   2> 141971 INFO  (TEST-TestLBHttpSolrClient.testTwoServers-seed#[67CE578726E381E8]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1901287845
   [junit4]   2> 141978 INFO  (TEST-TestLBHttpSolrClient.testTwoServers-seed#[67CE578726E381E8]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 141978 INFO  (TEST-TestLBHttpSolrClient.testTwoServers-seed#[67CE578726E381E8]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 141978 INFO  (TEST-TestLBHttpSolrClient.testTwoServers-seed#[67CE578726E381E8]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 141988 INFO  (coreCloseExecutor-166-thread-1) [    ] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@484277e5
   [junit4]   2> 141988 INFO  (coreCloseExecutor-166-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1212315621
   [junit4]   2> 142071 INFO  (TEST-TestLBHttpSolrClient.testTwoServers-seed#[67CE578726E381E8]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5a448c0d{/solr,null,UNAVAILABLE}
   [junit4]   2> 142123 INFO  (TEST-TestLBHttpSolrClient.testTwoServers-seed#[67CE578726E381E8]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@4444efb0{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 142123 INFO  (TEST-TestLBHttpSolrClient.testTwoServers-seed#[67CE578726E381E8]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=843680883
   [junit4]   2> 142123 INFO  (TEST-TestLBHttpSolrClient.testTwoServers-seed#[67CE578726E381E8]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 142123 INFO  (TEST-TestLBHttpSolrClient.testTwoServers-seed#[67CE578726E381E8]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 142123 INFO  (TEST-TestLBHttpSolrClient.testTwoServers-seed#[67CE578726E381E8]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 142126 INFO  (TEST-TestLBHttpSolrClient.testTwoServers-seed#[67CE578726E381E8]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 142127 INFO  (coreCloseExecutor-168-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5b131295
   [junit4]   2> 142127 INFO  (coreCloseExecutor-168-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1527976597
   [junit4]   2> 142227 INFO  (TEST-TestLBHttpSolrClient.testTwoServers-seed#[67CE578726E381E8]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1ba2f657{/solr,null,UNAVAILABLE}
   [junit4]   2> 142240 INFO  (TEST-TestLBHttpSolrClient.testTwoServers-seed#[67CE578726E381E8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testTwoServers
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrClient_67CE578726E381E8-001
   [junit4]   2> Sep 20, 2017 5:32:45 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {name=BlockTreeOrds(blocksize=128), id=BlockTreeOrds(blocksize=128)}, docValues:{}, maxPointsInLeafNode=661, maxMBSortInHeap=5.064963659438383, sim=RandomSimilarity(queryNorm=false): {}, locale=sr-Latn-BA, timezone=Asia/Makassar
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=2,free=266947448,total=321388544
   [junit4]   2> NOTE: All tests run in this JVM: [BasicHttpSolrClientTest, TestLBHttpSolrClient]
   [junit4] Completed [14/145 (1!)] on J0 in 59.44s, 3 tests, 1 failure <<< FAILURES!

[...truncated 45677 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-master - Build # 2104 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/2104/

11 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest.testAsyncRequests

Error Message:
Could not load collection from ZK: testAsyncOperations

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: testAsyncOperations
        at __randomizedtesting.SeedInfo.seed([D37EE6574257304F:373ADAE0E4FF7E90]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1115)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:128)
        at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:108)
        at org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest.testAsyncRequests(CollectionsAPIAsyncDistributedZkTest.java:128)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/testAsyncOperations/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1127)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1113)
        ... 43 more


FAILED:  org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter

Error Message:
Error from server at https://127.0.0.1:38102/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000016

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:38102/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000016
        at __randomizedtesting.SeedInfo.seed([D37EE6574257304F:862E0EC5EEAEFFBF]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter(CustomCollectionTest.java:135)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.RecoveryZkTest.test

Error Message:


Stack Trace:
java.util.concurrent.TimeoutException
        at __randomizedtesting.SeedInfo.seed([D37EE6574257304F:5B2AD98DECAB5DB7]:0)
        at org.apache.solr.common.cloud.ZkStateReader.waitForState(ZkStateReader.java:1268)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.waitForState(CloudSolrClient.java:438)
        at org.apache.solr.cloud.RecoveryZkTest.test(RecoveryZkTest.java:122)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestCloudRecovery.leaderRecoverFromLogOnStartupTest

Error Message:
expected:<2> but was:<3>

Stack Trace:
java.lang.AssertionError: expected:<2> but was:<3>
        at __randomizedtesting.SeedInfo.seed([D37EE6574257304F:A78E0712067377C0]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.cloud.TestCloudRecovery.leaderRecoverFromLogOnStartupTest(TestCloudRecovery.java:132)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestLocalFSCloudBackupRestore.test

Error Message:
expected:<COMPLETED> but was:<RUNNING>

Stack Trace:
java.lang.AssertionError: expected:<COMPLETED> but was:<RUNNING>
        at __randomizedtesting.SeedInfo.seed([D37EE6574257304F:5B2AD98DECAB5DB7]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:277)
        at org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:136)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestPullReplica.testRemoveAllWriterReplicas

Error Message:
Replica core_node4 not up to date after 10 seconds expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: Replica core_node4 not up to date after 10 seconds expected:<1> but was:<0>
        at __randomizedtesting.SeedInfo.seed([D37EE6574257304F:40707A6C56A9D9E6]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.TestPullReplica.waitForNumDocsInAllReplicas(TestPullReplica.java:510)
        at org.apache.solr.cloud.TestPullReplica.waitForNumDocsInAllReplicas(TestPullReplica.java:501)
        at org.apache.solr.cloud.TestPullReplica.doTestNoLeader(TestPullReplica.java:385)
        at org.apache.solr.cloud.TestPullReplica.testRemoveAllWriterReplicas(TestPullReplica.java:286)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestSolrConfigHandlerConcurrent.test

Error Message:
Could not load collection from ZK: collection1

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at __randomizedtesting.SeedInfo.seed([D37EE6574257304F:5B2AD98DECAB5DB7]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1115)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:128)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.getTotalReplicas(AbstractFullDistribZkTestBase.java:488)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:441)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:334)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1127)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1113)
        ... 38 more


FAILED:  org.apache.solr.update.TestInPlaceUpdatesDistrib.test

Error Message:
inplace_updatable_float didn't match for replica at client: https://127.0.0.1:37233/collection1 expected:<105.0> but was:<1.0>

Stack Trace:
java.lang.AssertionError: inplace_updatable_float didn't match for replica at client: https://127.0.0.1:37233/collection1 expected:<105.0> but was:<1.0>
        at __randomizedtesting.SeedInfo.seed([D37EE6574257304F:5B2AD98DECAB5DB7]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.assertReplicaValue(TestInPlaceUpdatesDistrib.java:980)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.outOfOrderUpdatesIndividualReplicaTest(TestInPlaceUpdatesDistrib.java:643)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:144)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.common.cloud.TestCollectionStateWatchers.testSimpleCollectionWatch

Error Message:
Could not load collection from ZK: testcollection

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: testcollection
        at __randomizedtesting.SeedInfo.seed([641C592B70F0F21E:3927965B37FD6D20]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1115)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:128)
        at org.apache.solr.common.cloud.ZkStateReader.registerCollectionStateWatcher(ZkStateReader.java:1233)
        at org.apache.solr.common.cloud.ZkStateReader.waitForState(ZkStateReader.java:1263)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.waitForState(CloudSolrClient.java:438)
        at org.apache.solr.common.cloud.TestCollectionStateWatchers.testSimpleCollectionWatch(TestCollectionStateWatchers.java:117)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/testcollection/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1127)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1113)
        ... 45 more


FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
Expected 2 of 3 replicas to be active but only found 1; [core_node4:{"core":"c8n_1x3_lf_shard1_replica_t1","base_url":"http://127.0.0.1:60090","node_name":"127.0.0.1:60090_","state":"active","type":"TLOG","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/16)={   "pullReplicas":"0",   "replicationFactor":"1",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node4":{           "core":"c8n_1x3_lf_shard1_replica_t1",           "base_url":"http://127.0.0.1:60090",           "node_name":"127.0.0.1:60090_",           "state":"active",           "type":"TLOG",           "leader":"true"},         "core_node5":{           "core":"c8n_1x3_lf_shard1_replica_t3",           "base_url":"http://127.0.0.1:51606",           "node_name":"127.0.0.1:51606_",           "state":"down",           "type":"TLOG"},         "core_node6":{           "state":"down",           "base_url":"http://127.0.0.1:58188",           "core":"c8n_1x3_lf_shard1_replica_t2",           "node_name":"127.0.0.1:58188_",           "type":"TLOG"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"0",   "tlogReplicas":"3"}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node4:{"core":"c8n_1x3_lf_shard1_replica_t1","base_url":"http://127.0.0.1:60090","node_name":"127.0.0.1:60090_","state":"active","type":"TLOG","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/16)={
  "pullReplicas":"0",
  "replicationFactor":"1",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node4":{
          "core":"c8n_1x3_lf_shard1_replica_t1",
          "base_url":"http://127.0.0.1:60090",
          "node_name":"127.0.0.1:60090_",
          "state":"active",
          "type":"TLOG",
          "leader":"true"},
        "core_node5":{
          "core":"c8n_1x3_lf_shard1_replica_t3",
          "base_url":"http://127.0.0.1:51606",
          "node_name":"127.0.0.1:51606_",
          "state":"down",
          "type":"TLOG"},
        "core_node6":{
          "state":"down",
          "base_url":"http://127.0.0.1:58188",
          "core":"c8n_1x3_lf_shard1_replica_t2",
          "node_name":"127.0.0.1:58188_",
          "type":"TLOG"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"0",
  "tlogReplicas":"3"}
        at __randomizedtesting.SeedInfo.seed([D37EE6574257304F:5B2AD98DECAB5DB7]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:169)
        at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:56)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
Didn't see all replicas for shard shard1 in c8n_1x2 come up within 90000 ms! ClusterState: {   "collMinRf_1x3":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{"shard1":{         "range":"80000000-7fffffff",         "state":"active",         "replicas":{           "core_node4":{             "core":"collMinRf_1x3_shard1_replica_t1",             "base_url":"http://127.0.0.1:54417/ui_sk/gi",             "node_name":"127.0.0.1:54417_ui_sk%2Fgi",             "state":"active",             "type":"TLOG",             "leader":"true"},           "core_node5":{             "core":"collMinRf_1x3_shard1_replica_t2",             "base_url":"http://127.0.0.1:58034/ui_sk/gi",             "node_name":"127.0.0.1:58034_ui_sk%2Fgi",             "state":"active",             "type":"TLOG"},           "core_node6":{             "core":"collMinRf_1x3_shard1_replica_t3",             "base_url":"http://127.0.0.1:50277/ui_sk/gi",             "node_name":"127.0.0.1:50277_ui_sk%2Fgi",             "state":"active",             "type":"TLOG"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"0",     "tlogReplicas":"3"},   "collection1":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{       "shard1":{         "range":"80000000-ffffffff",         "state":"active",         "replicas":{"core_node44":{             "core":"collection1_shard1_replica_t43",             "base_url":"http://127.0.0.1:58034/ui_sk/gi",             "node_name":"127.0.0.1:58034_ui_sk%2Fgi",             "state":"active",             "type":"TLOG",             "leader":"true"}}},       "shard2":{         "range":"0-7fffffff",         "state":"active",         "replicas":{           "core_node42":{             "core":"collection1_shard2_replica_t41",             "base_url":"http://127.0.0.1:56924/ui_sk/gi",             "node_name":"127.0.0.1:56924_ui_sk%2Fgi",             "state":"active",             "type":"TLOG",             "leader":"true"},           "core_node46":{             "core":"collection1_shard2_replica_t45",             "base_url":"http://127.0.0.1:54417/ui_sk/gi",             "node_name":"127.0.0.1:54417_ui_sk%2Fgi",             "state":"active",             "type":"TLOG"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"1",     "tlogReplicas":"0"},   "control_collection":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{"shard1":{         "range":"80000000-7fffffff",         "state":"active",         "replicas":{"core_node2":{             "core":"control_collection_shard1_replica_n1",             "base_url":"http://127.0.0.1:50277/ui_sk/gi",             "node_name":"127.0.0.1:50277_ui_sk%2Fgi",             "state":"active",             "type":"NRT",             "leader":"true"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"1",     "tlogReplicas":"0"},   "c8n_1x2":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{"shard1":{         "range":"80000000-7fffffff",         "state":"active",         "replicas":{           "core_node3":{             "state":"down",             "base_url":"http://127.0.0.1:54417/ui_sk/gi",             "core":"c8n_1x2_shard1_replica_t1",             "node_name":"127.0.0.1:54417_ui_sk%2Fgi",             "type":"TLOG"},           "core_node4":{             "core":"c8n_1x2_shard1_replica_t2",             "base_url":"http://127.0.0.1:50277/ui_sk/gi",             "node_name":"127.0.0.1:50277_ui_sk%2Fgi",             "state":"active",             "type":"TLOG",             "leader":"true"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"0",     "tlogReplicas":"2"}}

Stack Trace:
java.lang.AssertionError: Didn't see all replicas for shard shard1 in c8n_1x2 come up within 90000 ms! ClusterState: {
  "collMinRf_1x3":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{
          "core_node4":{
            "core":"collMinRf_1x3_shard1_replica_t1",
            "base_url":"http://127.0.0.1:54417/ui_sk/gi",
            "node_name":"127.0.0.1:54417_ui_sk%2Fgi",
            "state":"active",
            "type":"TLOG",
            "leader":"true"},
          "core_node5":{
            "core":"collMinRf_1x3_shard1_replica_t2",
            "base_url":"http://127.0.0.1:58034/ui_sk/gi",
            "node_name":"127.0.0.1:58034_ui_sk%2Fgi",
            "state":"active",
            "type":"TLOG"},
          "core_node6":{
            "core":"collMinRf_1x3_shard1_replica_t3",
            "base_url":"http://127.0.0.1:50277/ui_sk/gi",
            "node_name":"127.0.0.1:50277_ui_sk%2Fgi",
            "state":"active",
            "type":"TLOG"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"0",
    "tlogReplicas":"3"},
  "collection1":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{
      "shard1":{
        "range":"80000000-ffffffff",
        "state":"active",
        "replicas":{"core_node44":{
            "core":"collection1_shard1_replica_t43",
            "base_url":"http://127.0.0.1:58034/ui_sk/gi",
            "node_name":"127.0.0.1:58034_ui_sk%2Fgi",
            "state":"active",
            "type":"TLOG",
            "leader":"true"}}},
      "shard2":{
        "range":"0-7fffffff",
        "state":"active",
        "replicas":{
          "core_node42":{
            "core":"collection1_shard2_replica_t41",
            "base_url":"http://127.0.0.1:56924/ui_sk/gi",
            "node_name":"127.0.0.1:56924_ui_sk%2Fgi",
            "state":"active",
            "type":"TLOG",
            "leader":"true"},
          "core_node46":{
            "core":"collection1_shard2_replica_t45",
            "base_url":"http://127.0.0.1:54417/ui_sk/gi",
            "node_name":"127.0.0.1:54417_ui_sk%2Fgi",
            "state":"active",
            "type":"TLOG"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"1",
    "tlogReplicas":"0"},
  "control_collection":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{"core_node2":{
            "core":"control_collection_shard1_replica_n1",
            "base_url":"http://127.0.0.1:50277/ui_sk/gi",
            "node_name":"127.0.0.1:50277_ui_sk%2Fgi",
            "state":"active",
            "type":"NRT",
            "leader":"true"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"1",
    "tlogReplicas":"0"},
  "c8n_1x2":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{
          "core_node3":{
            "state":"down",
            "base_url":"http://127.0.0.1:54417/ui_sk/gi",
            "core":"c8n_1x2_shard1_replica_t1",
            "node_name":"127.0.0.1:54417_ui_sk%2Fgi",
            "type":"TLOG"},
          "core_node4":{
            "core":"c8n_1x2_shard1_replica_t2",
            "base_url":"http://127.0.0.1:50277/ui_sk/gi",
            "node_name":"127.0.0.1:50277_ui_sk%2Fgi",
            "state":"active",
            "type":"TLOG",
            "leader":"true"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"0",
    "tlogReplicas":"2"}}
        at __randomizedtesting.SeedInfo.seed([D37EE6574257304F:5B2AD98DECAB5DB7]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.ensureAllReplicasAreActive(AbstractFullDistribZkTestBase.java:1996)
        at org.apache.solr.cloud.HttpPartitionTest.testRf2(HttpPartitionTest.java:324)
        at org.apache.solr.cloud.HttpPartitionTest.test(HttpPartitionTest.java:132)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11732 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestPullReplica
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_D37EE6574257304F-001/init-core-data-001
   [junit4]   2> 783050 WARN  (SUITE-TestPullReplica-seed#[D37EE6574257304F]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=11 numCloses=11
   [junit4]   2> 783050 INFO  (SUITE-TestPullReplica-seed#[D37EE6574257304F]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 783051 INFO  (SUITE-TestPullReplica-seed#[D37EE6574257304F]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 783051 INFO  (SUITE-TestPullReplica-seed#[D37EE6574257304F]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 783134 INFO  (SUITE-TestPullReplica-seed#[D37EE6574257304F]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_D37EE6574257304F-001/tempDir-001
   [junit4]   2> 783137 INFO  (SUITE-TestPullReplica-seed#[D37EE6574257304F]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 783137 INFO  (Thread-274) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 783137 INFO  (Thread-274) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 783226 ERROR (Thread-274) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 783242 INFO  (SUITE-TestPullReplica-seed#[D37EE6574257304F]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:53801
   [junit4]   2> 783559 INFO  (jetty-launcher-179-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 783670 INFO  (jetty-launcher-179-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 784010 INFO  (jetty-launcher-179-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3fd319f5{/solr,null,AVAILABLE}
   [junit4]   2> 784021 INFO  (jetty-launcher-179-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4b6627e1{/solr,null,AVAILABLE}
   [junit4]   2> 784022 INFO  (jetty-launcher-179-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@589712e8{HTTP/1.1,[http/1.1]}{127.0.0.1:46528}
   [junit4]   2> 784022 INFO  (jetty-launcher-179-thread-2) [    ] o.e.j.s.Server Started @802361ms
   [junit4]   2> 784022 INFO  (jetty-launcher-179-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46528}
   [junit4]   2> 784023 ERROR (jetty-launcher-179-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 784023 INFO  (jetty-launcher-179-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 784023 INFO  (jetty-launcher-179-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 784023 INFO  (jetty-launcher-179-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 784023 INFO  (jetty-launcher-179-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-24T00:42:23.999Z
   [junit4]   2> 784023 INFO  (jetty-launcher-179-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@462779e2{HTTP/1.1,[http/1.1]}{127.0.0.1:49079}
   [junit4]   2> 784023 INFO  (jetty-launcher-179-thread-1) [    ] o.e.j.s.Server Started @802362ms
   [junit4]   2> 784023 INFO  (jetty-launcher-179-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=49079}
   [junit4]   2> 784023 ERROR (jetty-launcher-179-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 784023 INFO  (jetty-launcher-179-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 784023 INFO  (jetty-launcher-179-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 784023 INFO  (jetty-launcher-179-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 784023 INFO  (jetty-launcher-179-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-24T00:42:23.999Z
   [junit4]   2> 784226 INFO  (jetty-launcher-179-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 784312 INFO  (jetty-launcher-179-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53801/solr
   [junit4]   2> 784419 INFO  (jetty-launcher-179-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 784492 INFO  (jetty-launcher-179-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53801/solr
   [junit4]   2> 785121 INFO  (jetty-launcher-179-thread-2) [n:127.0.0.1:46528_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 785122 INFO  (jetty-launcher-179-thread-2) [n:127.0.0.1:46528_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46528_solr
   [junit4]   2> 785123 INFO  (jetty-launcher-179-thread-2) [n:127.0.0.1:46528_solr    ] o.a.s.c.Overseer Overseer (id=98711223871012870-127.0.0.1:46528_solr-n_0000000000) starting
   [junit4]   2> 785180 INFO  (jetty-launcher-179-thread-2) [n:127.0.0.1:46528_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46528_solr
   [junit4]   2> 785274 INFO  (OverseerStateUpdate-98711223871012870-127.0.0.1:46528_solr-n_0000000000) [n:127.0.0.1:46528_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 785414 INFO  (jetty-launcher-179-thread-1) [n:127.0.0.1:49079_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 785419 INFO  (jetty-launcher-179-thread-1) [n:127.0.0.1:49079_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 785430 INFO  (jetty-launcher-179-thread-1) [n:127.0.0.1:49079_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49079_solr
   [junit4]   2> 785434 INFO  (zkCallback-192-thread-1-processing-n:127.0.0.1:46528_solr) [n:127.0.0.1:46528_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 785577 INFO  (zkCallback-187-thread-1-processing-n:127.0.0.1:49079_solr) [n:127.0.0.1:49079_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 788127 INFO  (jetty-launcher-179-thread-2) [n:127.0.0.1:46528_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46528.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@59b90e06
   [junit4]   2> 788133 INFO  (jetty-launcher-179-thread-2) [n:127.0.0.1:46528_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46528.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@59b90e06
   [junit4]   2> 788134 INFO  (jetty-launcher-179-thread-2) [n:127.0.0.1:46528_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46528.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@59b90e06
   [junit4]   2> 788135 INFO  (jetty-launcher-179-thread-2) [n:127.0.0.1:46528_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_D37EE6574257304F-001/tempDir-001/node2/.
   [junit4]   2> 788408 INFO  (jetty-launcher-179-thread-1) [n:127.0.0.1:49079_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49079.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@59b90e06
   [junit4]   2> 788514 INFO  (jetty-launcher-179-thread-1) [n:127.0.0.1:49079_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49079.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@59b90e06
   [junit4]   2> 788515 INFO  (jetty-launcher-179-thread-1) [n:127.0.0.1:49079_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49079.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@59b90e06
   [junit4]   2> 788537 INFO  (jetty-launcher-179-thread-1) [n:127.0.0.1:49079_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_D37EE6574257304F-001/tempDir-001/node1/.
   [junit4]   2> 789330 INFO  (SUITE-TestPullReplica-seed#[D37EE6574257304F]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 789333 INFO  (SUITE-TestPullReplica-seed#[D37EE6574257304F]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:53801/solr ready
   [junit4]   2> 789368 INFO  (SUITE-TestPullReplica-seed#[D37EE6574257304F]-worker) [    ] o.a.s.c.TestPullReplica Using legacyCloud?: false
   [junit4]   2> 789402 INFO  (qtp840288847-1315) [n:127.0.0.1:46528_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterprop with params val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 789413 INFO  (qtp840288847-1315) [n:127.0.0.1:46528_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2} status=0 QTime=11
   [junit4]   2> 789486 INFO  (TEST-TestPullReplica.testKillPullReplica-seed#[D37EE6574257304F]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testKillPullReplica
   [junit4]   2> 789488 INFO  (qtp840288847-1316) [n:127.0.0.1:46528_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=1&replicationFactor=1&collection.configName=conf&maxShardsPerNode=100&name=pull_replica_test_kill_pull_replica&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 789529 INFO  (OverseerThreadFactory-570-thread-1-processing-n:127.0.0.1:46528_solr) [n:127.0.0.1:46528_solr    ] o.a.s.c.CreateCollectionCmd Create collection pull_replica_test_kill_pull_replica
   [junit4]   2> 789778 INFO  (OverseerStateUpdate-98711223871012870-127.0.0.1:46528_solr-n_0000000000) [n:127.0.0.1:46528_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"pull_replica_test_kill_pull_replica",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"pull_replica_test_kill_pull_replica_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:46528/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 789779 INFO  (OverseerStateUpdate-98711223871012870-127.0.0.1:46528_solr-n_0000000000) [n:127.0.0.1:46528_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"pull_replica_test_kill_pull_replica",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"pull_replica_test_kill_pull_replica_shard1_replica_p2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:49079/solr",
   [junit4]   2>   "type":"PULL"}
   [junit4]   2> 790322 INFO  (qtp840288847-1313) [n:127.0.0.1:46528_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=pull_replica_test_kill_pull_replica_shard1_replica_n1&action=CREATE&numShards=1&collection=pull_replica_test_kill_pull_replica&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 790323 INFO  (qtp840288847-1313) [n:127.0.0.1:46528_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 790492 INFO  (zkCallback-192-thread-1-processing-n:127.0.0.1:46528_solr) [n:127.0.0.1:46528_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 791015 INFO  (qtp440333781-1306) [n:127.0.0.1:49079_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=pull_replica_test_kill_pull_replica_shard1_replica_p2&action=CREATE&numShards=1&collection=pull_replica_test_kill_pull_replica&shard=shard1&wt=javabin&version=2&replicaType=PULL
   [junit4]   2> 791046 INFO  (qtp440333781-1306) [n:127.0.0.1:49079_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 790969 INFO  (zkCallback-192-thread-2-processing-n:127.0.0.1:46528_solr) [n:127.0.0.1:46528_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 791178 INFO  (zkCallback-187-thread-1-processing-n:127.0.0.1:49079_solr) [n:127.0.0.1:49079_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 791194 INFO  (zkCallback-187-thread-2-processing-n:127.0.0.1:49079_solr) [n:127.0.0.1:49079_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 791248 INFO  (zkCallback-192-thread-2-processing-n:127.0.0.1:46528_solr) [n:127.0.0.1:46528_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 791249 INFO  (zkCallback-192-thread-3-processing-n:127.0.0.1:46528_solr) [n:127.0.0.1:46528_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 791990 INFO  (qtp840288847-1313) [n:127.0.0.1:46528_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 792144 INFO  (qtp840288847-1313) [n:127.0.0.1:46528_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.s.IndexSchema [pull_replica_test_kill_pull_replica_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 792183 INFO  (qtp840288847-1313) [n:127.0.0.1:46528_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 792183 INFO  (qtp840288847-1313) [n:127.0.0.1:46528_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'pull_replica_test_kill_pull_replica_shard1_replica_n1' using configuration from collection pull_replica_test_kill_pull_replica, trusted=true
   [junit4]   2> 792183 INFO  (qtp840288847-1313) [n:127.0.0.1:46528_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46528.solr.core.pull_replica_test_kill_pull_replica.shard1.replica_n1' (registry 'solr.core.pull_replica_test_kill_pull_replica.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@59b90e06
   [junit4]   2> 792183 INFO  (qtp840288847-1313) [n:127.0.0.1:46528_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 792183 INFO  (qtp840288847-1313) [n:127.0.0.1:46528_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.SolrCore [[pull_replica_test_kill_pull_replica_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_D37EE6574257304F-001/tempDir-001/node2/pull_replica_test_kill_pull_replica_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_D37EE6574257304F-001/tempDir-001/node2/./pull_replica_test_kill_pull_replica_shard1_replica_n1/data/]
   [junit4]   2> 792568 INFO  (qtp440333781-1306) [n:127.0.0.1:49079_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 792632 INFO  (qtp440333781-1306) [n:127.0.0.1:49079_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.s.IndexSchema [pull_replica_test_kill_pull_replica_shard1_replica_p2] Schema name=minimal
   [junit4]   2> 792690 INFO  (qtp440333781-1306) [n:127.0.0.1:49079_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 792690 INFO  (qtp440333781-1306) [n:127.0.0.1:49079_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.CoreContainer Creating SolrCore 'pull_replica_test_kill_pull_replica_shard1_replica_p2' using configuration from collection pull_replica_test_kill_pull_replica, trusted=true
   [junit4]   2> 792690 INFO  (qtp440333781-1306) [n:127.0.0.1:49079_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49079.solr.core.pull_replica_test_kill_pull_replica.shard1.replica_p2' (registry 'solr.core.pull_replica_test_kill_pull_replica.shard1.replica_p2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@59b90e06
   [junit4]   2> 792691 INFO  (qtp440333781-1306) [n:127.0.0.1:49079_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 792691 INFO  (qtp440333781-1306) [n:127.0.0.1:49079_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.SolrCore [[pull_replica_test_kill_pull_replica_shard1_replica_p2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_D37EE6574257304F-001/tempDir-001/node1/pull_replica_test_kill_pull_replica_shard1_replica_p2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_D37EE6574257304F-001/tempDir-001/node1/./pull_replica_test_kill_pull_replica_shard1_replica_p2/data/]
   [junit4]   2> 793257 INFO  (qtp840288847-1313) [n:127.0.0.1:46528_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 793257 INFO  (qtp840288847-1313) [n:127.0.0.1:46528_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 793355 INFO  (qtp840288847-1313) [n:127.0.0.1:46528_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 793355 INFO  (qtp840288847-1313) [n:127.0.0.1:46528_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 793453 INFO  (qtp440333781-1306) [n:127.0.0.1:49079_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 793453 INFO  (qtp440333781-1306) [n:127.0.0.1:49079_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 793462 INFO  (qtp840288847-1313) [n:127.0.0.1:46528_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1f3b0647[pull_replica_test_kill_pull_replica_shard1_replica_n1] main]
   [junit4]   2> 793462 INFO  (qtp440333781-1306) [n:127.0.0.1:49079_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.s.SolrIndexSearcher Opening [Searcher@799682b7[pull_replica_test_kill_pull_replica_shard1_replica_p2] main]
   [junit4]   2> 793466 INFO  (qtp440333781-1306) [n:127.0.0.1:49079_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 793466 INFO  (qtp440333781-1306) [n:127.0.0.1:49079_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 793467 INFO  (qtp440333781-1306) [n:127.0.0.1:49079_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 793470 INFO  (qtp840288847-1313) [n:127.0.0.1:46528_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 793471 INFO  (qtp840288847-1313) [n:127.0.0.1:46528_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 793472 INFO  (qtp840288847-1313) [n:127.0.0.1:46528_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 793536 INFO  (searcherExecutor-575-thread-1-processing-n:127.0.0.1:46528_solr x:pull_replica_test_kill_pull_replica_shard1_replica_n1 s:shard1 c:pull_replica_test_kill_pull_replica r:core_node3) [n:127.0.0.1:46528_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.SolrCore [pull_replica_test_kill_pull_replica_shard1_replica_n1] Registered new searcher Searcher@1f3b0647[pull_replica_test_kill_pull_replica_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 793537 INFO  (searcherExecutor-576-thread-1-processing-n:127.0.0.1:49079_solr x:pull_replica_test_kill_pull_replica_shard1_replica_p2 s:shard1 c:pull_replica_test_kill_pull_replica r:core_node4) [n:127.0.0.1:49079_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.SolrCore [pull_replica_test_kill_pull_replica_shard1_replica_p2] Registered new searcher Searcher@799682b7[pull_replica_test_kill_pull_replica_shard1_replica_p2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 793538 INFO  (qtp440333781-1306) [n:127.0.0.1:49079_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.ZkController pull_replica_test_kill_pull_replica_shard1_replica_p2 starting background replication from leader
   [junit4]   2> 793538 INFO  (qtp440333781-1306) [n:127.0.0.1:49079

[...truncated too long message...]

tateformat1_shard1_replica_n1] o.a.s.c.SolrCore [stateformat1_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5ded8c51
   [junit4]   2> 186927 INFO  (coreCloseExecutor-129-thread-2) [n:127.0.0.1:38516_solr c:testcollection s:shard4 r:core_node8 x:testcollection_shard4_replica_n7] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testcollection.shard4.leader, tag=40979878
   [junit4]   2> 186957 INFO  (coreCloseExecutor-129-thread-1) [n:127.0.0.1:38516_solr c:falsepredicate s:shard4 r:core_node8 x:falsepredicate_shard4_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.falsepredicate.shard4.leader, tag=1106305326
   [junit4]   2> 186986 INFO  (coreCloseExecutor-131-thread-4) [n:127.0.0.1:59494_solr c:falsepredicate s:shard1 r:core_node3 x:falsepredicate_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.falsepredicate.shard1.replica_n1, tag=1350282811
   [junit4]   2> 186986 INFO  (coreCloseExecutor-131-thread-4) [n:127.0.0.1:59494_solr c:falsepredicate s:shard1 r:core_node3 x:falsepredicate_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@58dae8ee: rootName = solr_59494, domain = solr.core.falsepredicate.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.falsepredicate.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@4fcef3fa
   [junit4]   2> 187009 INFO  (coreCloseExecutor-130-thread-1) [n:127.0.0.1:45348_solr c:testcollection s:shard1 r:core_node2 x:testcollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testcollection.shard1.leader, tag=1412877550
   [junit4]   2> 187009 INFO  (coreCloseExecutor-130-thread-3) [n:127.0.0.1:45348_solr c:currentstate s:shard1 r:core_node2 x:currentstate_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.currentstate.shard1.leader, tag=1456421059
   [junit4]   2> 187677 INFO  (zkCallback-24-thread-1-processing-n:127.0.0.1:59494_solr) [n:127.0.0.1:59494_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 187677 INFO  (zkCallback-21-thread-2-processing-n:127.0.0.1:45348_solr) [n:127.0.0.1:45348_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 187677 INFO  (zkCallback-48-thread-4-processing-n:127.0.0.1:38516_solr) [n:127.0.0.1:38516_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 187719 INFO  (coreCloseExecutor-131-thread-2) [n:127.0.0.1:59494_solr c:waitforstate s:shard1 r:core_node2 x:waitforstate_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.waitforstate.shard1.leader, tag=320046093
   [junit4]   2> 187719 INFO  (coreCloseExecutor-131-thread-4) [n:127.0.0.1:59494_solr c:falsepredicate s:shard1 r:core_node3 x:falsepredicate_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.falsepredicate.shard1.leader, tag=1350282811
   [junit4]   2> 187787 INFO  (coreCloseExecutor-131-thread-5) [n:127.0.0.1:59494_solr c:stateformat1 s:shard1 r:core_node2 x:stateformat1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.stateformat1.shard1.replica_n1, tag=1575849041
   [junit4]   2> 187787 INFO  (coreCloseExecutor-131-thread-5) [n:127.0.0.1:59494_solr c:stateformat1 s:shard1 r:core_node2 x:stateformat1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3a3fd59c: rootName = solr_59494, domain = solr.core.stateformat1.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.stateformat1.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@5e2df3b
   [junit4]   2> 188227 INFO  (zkCallback-24-thread-1-processing-n:127.0.0.1:59494_solr) [n:127.0.0.1:59494_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 188227 INFO  (zkCallback-21-thread-2-processing-n:127.0.0.1:45348_solr) [n:127.0.0.1:45348_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 188228 INFO  (coreCloseExecutor-131-thread-5) [n:127.0.0.1:59494_solr c:stateformat1 s:shard1 r:core_node2 x:stateformat1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.stateformat1.shard1.leader, tag=1575849041
   [junit4]   2> 188229 INFO  (coreCloseExecutor-131-thread-3) [n:127.0.0.1:59494_solr c:delayed s:shard1 r:core_node2 x:delayed_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.delayed.shard1.leader, tag=204287829
   [junit4]   2> 188442 INFO  (jetty-closer-3-thread-1) [    ] o.a.s.c.Overseer Overseer (id=98711908197203977-127.0.0.1:45348_solr-n_0000000000) closing
   [junit4]   2> 188442 INFO  (OverseerStateUpdate-98711908197203977-127.0.0.1:45348_solr-n_0000000000) [n:127.0.0.1:45348_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:45348_solr
   [junit4]   2> 189273 WARN  (zkCallback-42-thread-2-processing-n:127.0.0.1:57873_solr) [n:127.0.0.1:57873_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 190268 WARN  (zkCallback-48-thread-2-processing-n:127.0.0.1:38516_solr) [n:127.0.0.1:38516_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 190471 INFO  (jetty-closer-3-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5ea3ad46{/solr,null,UNAVAILABLE}
   [junit4]   2> 190492 INFO  (jetty-closer-3-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1c420d7b{/solr,null,UNAVAILABLE}
   [junit4]   2> 190494 INFO  (zkCallback-24-thread-3-processing-n:127.0.0.1:59494_solr) [n:127.0.0.1:59494_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59494_solr
   [junit4]   2> 190494 INFO  (zkCallback-24-thread-1-processing-n:127.0.0.1:59494_solr) [n:127.0.0.1:59494_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 192001 WARN  (zkCallback-21-thread-2-processing-n:127.0.0.1:45348_solr) [n:127.0.0.1:45348_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 192002 INFO  (jetty-closer-3-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@47bcbaf8{/solr,null,UNAVAILABLE}
   [junit4]   2> 192029 WARN  (zkCallback-24-thread-3-processing-n:127.0.0.1:59494_solr) [n:127.0.0.1:59494_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 192029 INFO  (jetty-closer-3-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@53531fe1{/solr,null,UNAVAILABLE}
   [junit4]   2> 192046 ERROR (SUITE-TestCollectionStateWatchers-seed#[641C592B70F0F21E]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 192047 INFO  (SUITE-TestCollectionStateWatchers-seed#[641C592B70F0F21E]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41683 41683
   [junit4]   2> 192219 INFO  (Thread-1) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41683 41683
   [junit4]   2> 192223 WARN  (Thread-1) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 8 /solr/aliases.json
   [junit4]   2> 8 /solr/clusterprops.json
   [junit4]   2> 6 /solr/security.json
   [junit4]   2> 6 /solr/configs/config
   [junit4]   2> 3 /solr/collections/stateformat1/state.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 79 /solr/collections/testcollection/state.json
   [junit4]   2> 73 /solr/collections/falsepredicate/state.json
   [junit4]   2> 18 /solr/collections/waitforstate/state.json
   [junit4]   2> 8 /solr/collections/delayed/state.json
   [junit4]   2> 8 /solr/clusterstate.json
   [junit4]   2> 7 /solr/collections/tobedeleted/state.json
   [junit4]   2> 5 /solr/collections/stateformat1/state.json
   [junit4]   2> 3 /solr/overseer_elect/election/98711908197203980-127.0.0.1:59494_solr-n_0000000001
   [junit4]   2> 2 /solr/collections/currentstate/state.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 8 /solr/collections
   [junit4]   2> 6 /solr/live_nodes
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_641C592B70F0F21E-001
   [junit4]   2> Sep 24, 2017 3:39:32 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1271, maxMBSortInHeap=5.373536901245038, sim=RandomSimilarity(queryNorm=false): {}, locale=da-DK, timezone=Etc/GMT+11
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=193536672,total=285736960
   [junit4]   2> NOTE: All tests run in this JVM: [TestCollectionStateWatchers]
   [junit4] Completed [21/145 (1!)] on J1 in 202.63s, 9 tests, 1 error <<< FAILURES!

[...truncated 46056 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-master - Build # 2105 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/2105/

10 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCreateNodeSet

Error Message:
KeeperErrorCode = Session expired for /clusterstate.json

Stack Trace:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
        at __randomizedtesting.SeedInfo.seed([A18CA5ECF70A4418:FA2F720C9E8B6090]:0)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
        at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:384)
        at org.apache.solr.cloud.MiniSolrCloudCluster.deleteAllCollections(MiniSolrCloudCluster.java:442)
        at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.clearCluster(CollectionsAPIDistributedZkTest.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:968)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.core.TestLazyCores.testNoCommit

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
        at __randomizedtesting.SeedInfo.seed([A18CA5ECF70A4418:7EEC043D3C2D27BD]:0)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:884)
        at org.apache.solr.core.TestLazyCores.check10(TestLazyCores.java:847)
        at org.apache.solr.core.TestLazyCores.testNoCommit(TestLazyCores.java:829)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound='10']
        xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int><lst name="params"><str name="q">*:*</str></lst></lst><result name="response" numFound="0" start="0"></result>
</response>

        request was:q=*:*
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:877)
        ... 41 more


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.core.TestLazyCores:     1) Thread[id=11375, name=searcherExecutor-3295-thread-1, state=WAITING, group=TGRP-TestLazyCores]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=11365, name=searcherExecutor-3279-thread-1, state=WAITING, group=TGRP-TestLazyCores]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.core.TestLazyCores:
   1) Thread[id=11375, name=searcherExecutor-3295-thread-1, state=WAITING, group=TGRP-TestLazyCores]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=11365, name=searcherExecutor-3279-thread-1, state=WAITING, group=TGRP-TestLazyCores]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([A18CA5ECF70A4418]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=11375, name=searcherExecutor-3295-thread-1, state=WAITING, group=TGRP-TestLazyCores]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=11365, name=searcherExecutor-3279-thread-1, state=WAITING, group=TGRP-TestLazyCores]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=11375, name=searcherExecutor-3295-thread-1, state=WAITING, group=TGRP-TestLazyCores]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=11365, name=searcherExecutor-3279-thread-1, state=WAITING, group=TGRP-TestLazyCores]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([A18CA5ECF70A4418]:0)


FAILED:  org.apache.solr.security.TestPKIAuthenticationPlugin.test

Error Message:


Stack Trace:
java.lang.NullPointerException
        at __randomizedtesting.SeedInfo.seed([A18CA5ECF70A4418:29D89A3659F629E0]:0)
        at org.apache.solr.security.TestPKIAuthenticationPlugin.test(TestPKIAuthenticationPlugin.java:100)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.io.graph.GraphExpressionTest.testGatherNodesFriendsStream

Error Message:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: org.apache.solr.common.SolrException: Could not load collection from ZK: collection1

Stack Trace:
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at __randomizedtesting.SeedInfo.seed([A1497D5BF4154CA7:A22505251935341C]:0)
        at org.apache.solr.client.solrj.io.graph.GatherNodesStream.read(GatherNodesStream.java:602)
        at org.apache.solr.client.solrj.io.stream.HashJoinStream.read(HashJoinStream.java:239)
        at org.apache.solr.client.solrj.io.graph.GraphExpressionTest.getTuples(GraphExpressionTest.java:932)
        at org.apache.solr.client.solrj.io.graph.GraphExpressionTest.testGatherNodesFriendsStream(GraphExpressionTest.java:748)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.solr.client.solrj.io.graph.GatherNodesStream.read(GatherNodesStream.java:574)
        ... 42 more
Caused by: java.lang.RuntimeException: java.io.IOException: org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at org.apache.solr.client.solrj.io.graph.GatherNodesStream$JoinRunner.call(GatherNodesStream.java:489)
        at org.apache.solr.client.solrj.io.graph.GatherNodesStream$JoinRunner.call(GatherNodesStream.java:408)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more
Caused by: java.io.IOException: org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at org.apache.solr.client.solrj.io.stream.CloudSolrStream.constructStreams(CloudSolrStream.java:383)
        at org.apache.solr.client.solrj.io.stream.CloudSolrStream.open(CloudSolrStream.java:274)
        at org.apache.solr.client.solrj.io.stream.PushBackStream.open(PushBackStream.java:70)
        at org.apache.solr.client.solrj.io.stream.ReducerStream.open(ReducerStream.java:199)
        at org.apache.solr.client.solrj.io.stream.UniqueStream.open(UniqueStream.java:150)
        at org.apache.solr.client.solrj.io.graph.GatherNodesStream$JoinRunner.call(GatherNodesStream.java:478)
        ... 6 more
Caused by: org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1115)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.common.cloud.ClusterState.getCollectionsMap(ClusterState.java:144)
        at org.apache.solr.client.solrj.io.stream.TupleStream.getSlices(TupleStream.java:167)
        at org.apache.solr.client.solrj.io.stream.TupleStream.getShards(TupleStream.java:141)
        at org.apache.solr.client.solrj.io.stream.CloudSolrStream.constructStreams(CloudSolrStream.java:368)
        ... 11 more
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1127)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1113)
        ... 16 more


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.graph.GraphExpressionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.client.solrj.io.graph.GraphExpressionTest:     1) Thread[id=1644, name=Connection evictor, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=1702, name=zkCallback-309-thread-2-EventThread, state=WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    3) Thread[id=1680, name=zkCallback-309-thread-3, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=1640, name=zkCallback-309-thread-1, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=1697, name=zkCallback-309-thread-4, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=1637, name=Connection evictor, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=1643, name=Connection evictor, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    8) Thread[id=1701, name=zkCallback-309-thread-2-SendThread(127.0.0.1:59361), state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    9) Thread[id=1679, name=zkCallback-309-thread-2, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   10) Thread[id=1797, name=zkCallback-309-thread-5, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.client.solrj.io.graph.GraphExpressionTest:
   1) Thread[id=1644, name=Connection evictor, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=1702, name=zkCallback-309-thread-2-EventThread, state=WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   3) Thread[id=1680, name=zkCallback-309-thread-3, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=1640, name=zkCallback-309-thread-1, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=1697, name=zkCallback-309-thread-4, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=1637, name=Connection evictor, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=1643, name=Connection evictor, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=1701, name=zkCallback-309-thread-2-SendThread(127.0.0.1:59361), state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   9) Thread[id=1679, name=zkCallback-309-thread-2, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  10) Thread[id=1797, name=zkCallback-309-thread-5, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([A1497D5BF4154CA7]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.graph.GraphExpressionTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=1680, name=zkCallback-309-thread-3, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=1640, name=zkCallback-309-thread-1, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=1697, name=zkCallback-309-thread-4, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=1701, name=zkCallback-309-thread-2-SendThread(127.0.0.1:59361), state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    5) Thread[id=1679, name=zkCallback-309-thread-2, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=1797, name=zkCallback-309-thread-5, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=1680, name=zkCallback-309-thread-3, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=1640, name=zkCallback-309-thread-1, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=1697, name=zkCallback-309-thread-4, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=1701, name=zkCallback-309-thread-2-SendThread(127.0.0.1:59361), state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   5) Thread[id=1679, name=zkCallback-309-thread-2, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=1797, name=zkCallback-309-thread-5, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([A1497D5BF4154CA7]:0)


FAILED:  org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testReverse

Error Message:
Error from server at http://127.0.0.1:35232/solr/collection1_shard2_replica_n2: Cannot talk to ZooKeeper - Updates are disabled.

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://127.0.0.1:35232/solr/collection1_shard2_replica_n2: Cannot talk to ZooKeeper - Updates are disabled.
        at __randomizedtesting.SeedInfo.seed([A1497D5BF4154CA7:B3F58026AB34FFA5]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:539)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:993)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
        at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testReverse(StreamExpressionTest.java:5658)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:35232/solr/collection1_shard2_replica_n2: Cannot talk to ZooKeeper - Updates are disabled.
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:516)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more


FAILED:  org.apache.solr.cloud.TestLocalFSCloudBackupRestore.test

Error Message:
expected:<COMPLETED> but was:<RUNNING>

Stack Trace:
java.lang.AssertionError: expected:<COMPLETED> but was:<RUNNING>
        at __randomizedtesting.SeedInfo.seed([A18CA5ECF70A4418:29D89A3659F629E0]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:277)
        at org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:136)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11582 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestLocalFSCloudBackupRestore
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.TestLocalFSCloudBackupRestore_A18CA5ECF70A4418-001/init-core-data-001
   [junit4]   2> 278167 WARN  (SUITE-TestLocalFSCloudBackupRestore-seed#[A18CA5ECF70A4418]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=30 numCloses=30
   [junit4]   2> 278167 INFO  (SUITE-TestLocalFSCloudBackupRestore-seed#[A18CA5ECF70A4418]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 278220 INFO  (SUITE-TestLocalFSCloudBackupRestore-seed#[A18CA5ECF70A4418]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 278221 INFO  (SUITE-TestLocalFSCloudBackupRestore-seed#[A18CA5ECF70A4418]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 278242 INFO  (SUITE-TestLocalFSCloudBackupRestore-seed#[A18CA5ECF70A4418]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.TestLocalFSCloudBackupRestore_A18CA5ECF70A4418-001/tempDir-001
   [junit4]   2> 278243 INFO  (SUITE-TestLocalFSCloudBackupRestore-seed#[A18CA5ECF70A4418]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 278268 INFO  (Thread-36) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 278268 INFO  (Thread-36) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 278365 ERROR (Thread-36) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 278371 INFO  (SUITE-TestLocalFSCloudBackupRestore-seed#[A18CA5ECF70A4418]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:34249
   [junit4]   2> 278517 INFO  (jetty-launcher-169-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 278517 INFO  (jetty-launcher-169-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 278557 INFO  (jetty-launcher-169-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@594b26fd{/solr,null,AVAILABLE}
   [junit4]   2> 278701 INFO  (jetty-launcher-169-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@303e9dff{/solr,null,AVAILABLE}
   [junit4]   2> 278801 INFO  (jetty-launcher-169-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1aaba3af{HTTP/1.1,[http/1.1]}{127.0.0.1:51096}
   [junit4]   2> 278802 INFO  (jetty-launcher-169-thread-1) [    ] o.e.j.s.Server Started @301707ms
   [junit4]   2> 278802 INFO  (jetty-launcher-169-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=51096}
   [junit4]   2> 278803 ERROR (jetty-launcher-169-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 278803 INFO  (jetty-launcher-169-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 278803 INFO  (jetty-launcher-169-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 278803 INFO  (jetty-launcher-169-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 278803 INFO  (jetty-launcher-169-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-26T15:56:58.432Z
   [junit4]   2> 278802 INFO  (jetty-launcher-169-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@50742ad5{HTTP/1.1,[http/1.1]}{127.0.0.1:34997}
   [junit4]   2> 278803 INFO  (jetty-launcher-169-thread-2) [    ] o.e.j.s.Server Started @301708ms
   [junit4]   2> 278803 INFO  (jetty-launcher-169-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 278803 INFO  (jetty-launcher-169-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34997}
   [junit4]   2> 278839 ERROR (jetty-launcher-169-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 278898 INFO  (jetty-launcher-169-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 278899 INFO  (jetty-launcher-169-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 278899 INFO  (jetty-launcher-169-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 278899 INFO  (jetty-launcher-169-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-26T15:56:58.528Z
   [junit4]   2> 278905 INFO  (jetty-launcher-169-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 279041 INFO  (jetty-launcher-169-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 279201 INFO  (jetty-launcher-169-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34249/solr
   [junit4]   2> 279385 INFO  (jetty-launcher-169-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34249/solr
   [junit4]   2> 279498 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ebee7d4aa0004, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 279829 INFO  (jetty-launcher-169-thread-1) [n:127.0.0.1:51096_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 279854 INFO  (jetty-launcher-169-thread-1) [n:127.0.0.1:51096_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:51096_solr
   [junit4]   2> 279854 INFO  (jetty-launcher-169-thread-1) [n:127.0.0.1:51096_solr    ] o.a.s.c.Overseer Overseer (id=98726144763363334-127.0.0.1:51096_solr-n_0000000000) starting
   [junit4]   2> 280020 INFO  (jetty-launcher-169-thread-1) [n:127.0.0.1:51096_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51096_solr
   [junit4]   2> 280120 INFO  (OverseerStateUpdate-98726144763363334-127.0.0.1:51096_solr-n_0000000000) [n:127.0.0.1:51096_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 280776 INFO  (jetty-launcher-169-thread-2) [n:127.0.0.1:34997_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 280778 INFO  (jetty-launcher-169-thread-2) [n:127.0.0.1:34997_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 280779 INFO  (jetty-launcher-169-thread-2) [n:127.0.0.1:34997_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34997_solr
   [junit4]   2> 280909 INFO  (zkCallback-181-thread-1-processing-n:127.0.0.1:51096_solr) [n:127.0.0.1:51096_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 281022 INFO  (zkCallback-180-thread-1-processing-n:127.0.0.1:34997_solr) [n:127.0.0.1:34997_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 282880 INFO  (jetty-launcher-169-thread-1) [n:127.0.0.1:51096_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51096.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6db954c1
   [junit4]   2> 282906 INFO  (jetty-launcher-169-thread-2) [n:127.0.0.1:34997_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34997.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6db954c1
   [junit4]   2> 283115 INFO  (jetty-launcher-169-thread-1) [n:127.0.0.1:51096_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51096.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6db954c1
   [junit4]   2> 283116 INFO  (jetty-launcher-169-thread-1) [n:127.0.0.1:51096_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51096.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6db954c1
   [junit4]   2> 284773 INFO  (jetty-launcher-169-thread-2) [n:127.0.0.1:34997_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34997.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6db954c1
   [junit4]   2> 284774 INFO  (jetty-launcher-169-thread-2) [n:127.0.0.1:34997_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34997.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6db954c1
   [junit4]   2> 284775 INFO  (jetty-launcher-169-thread-2) [n:127.0.0.1:34997_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.TestLocalFSCloudBackupRestore_A18CA5ECF70A4418-001/tempDir-001/node2/.
   [junit4]   2> 283294 INFO  (jetty-launcher-169-thread-1) [n:127.0.0.1:51096_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.TestLocalFSCloudBackupRestore_A18CA5ECF70A4418-001/tempDir-001/node1/.
   [junit4]   2> 285585 INFO  (SUITE-TestLocalFSCloudBackupRestore-seed#[A18CA5ECF70A4418]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 285645 INFO  (SUITE-TestLocalFSCloudBackupRestore-seed#[A18CA5ECF70A4418]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34249/solr ready
   [junit4]   2> 286029 INFO  (TEST-TestLocalFSCloudBackupRestore.test-seed#[A18CA5ECF70A4418]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 288329 INFO  (qtp605417064-531) [n:127.0.0.1:34997_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf1&router.name=implicit&version=2&pullReplicas=1&shards=shard1,shard2&property.customKey=customValue&maxShardsPerNode=4&router.field=shard_s&autoAddReplicas=true&name=backuprestore&nrtReplicas=2&action=CREATE&tlogReplicas=1&wt=javabin and sendToOCPQueue=true
   [junit4]   2> 288497 INFO  (OverseerThreadFactory-166-thread-1-processing-n:127.0.0.1:51096_solr) [n:127.0.0.1:51096_solr    ] o.a.s.c.CreateCollectionCmd Create collection backuprestore
   [junit4]   2> 288501 WARN  (OverseerThreadFactory-166-thread-1-processing-n:127.0.0.1:51096_solr) [n:127.0.0.1:51096_solr    ] o.a.s.c.CreateCollectionCmd Specified number of replicas of 4 on collection backuprestore is higher than the number of Solr instances currently live or live and part of your createNodeSet(2). It's unusual to run two replica of the same slice on the same Solr-instance.
   [junit4]   2> 288722 INFO  (OverseerStateUpdate-98726144763363334-127.0.0.1:51096_solr-n_0000000000) [n:127.0.0.1:51096_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"backuprestore",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"backuprestore_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:51096/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 288730 INFO  (OverseerStateUpdate-98726144763363334-127.0.0.1:51096_solr-n_0000000000) [n:127.0.0.1:51096_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"backuprestore",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"backuprestore_shard1_replica_n3",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:34997/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 288738 INFO  (OverseerStateUpdate-98726144763363334-127.0.0.1:51096_solr-n_0000000000) [n:127.0.0.1:51096_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"backuprestore",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"backuprestore_shard1_replica_t5",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:51096/solr",
   [junit4]   2>   "type":"TLOG"}
   [junit4]   2> 288759 INFO  (OverseerStateUpdate-98726144763363334-127.0.0.1:51096_solr-n_0000000000) [n:127.0.0.1:51096_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"backuprestore",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"backuprestore_shard1_replica_p6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:34997/solr",
   [junit4]   2>   "type":"PULL"}
   [junit4]   2> 288776 INFO  (OverseerStateUpdate-98726144763363334-127.0.0.1:51096_solr-n_0000000000) [n:127.0.0.1:51096_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"backuprestore",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"backuprestore_shard2_replica_n8",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:51096/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 288780 INFO  (OverseerStateUpdate-98726144763363334-127.0.0.1:51096_solr-n_0000000000) [n:127.0.0.1:51096_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"backuprestore",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"backuprestore_shard2_replica_n10",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:34997/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 288849 INFO  (OverseerStateUpdate-98726144763363334-127.0.0.1:51096_solr-n_0000000000) [n:127.0.0.1:51096_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"backuprestore",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"backuprestore_shard2_replica_t12",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:51096/solr",
   [junit4]   2>   "type":"TLOG"}
   [junit4]   2> 288851 INFO  (OverseerStateUpdate-98726144763363334-127.0.0.1:51096_solr-n_0000000000) [n:127.0.0.1:51096_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"backuprestore",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"backuprestore_shard2_replica_p14",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:34997/solr",
   [junit4]   2>   "type":"PULL"}
   [junit4]   2> 289881 INFO  (qtp605417064-528) [n:127.0.0.1:34997_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=backuprestore&version=2&replicaType=NRT&property.customKey=customValue&coreNodeName=core_node4&name=backuprestore_shard1_replica_n3&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 289881 INFO  (qtp605417064-528) [n:127.0.0.1:34997_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 289881 INFO  (qtp341142357-519) [n:127.0.0.1:51096_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=backuprestore&version=2&replicaType=TLOG&property.customKey=customValue&coreNodeName=core_node7&name=backuprestore_shard1_replica_t5&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 289963 INFO  (qtp341142357-519) [n:127.0.0.1:51096_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 289882 INFO  (qtp341142357-518) [n:127.0.0.1:51096_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=backuprestore&version=2&replicaType=NRT&property.customKey=customValue&coreNodeName=core_node11&name=backuprestore_shard2_replica_n8&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 289982 WARN  (OverseerHdfsCoreFailoverThread-98726144763363334-127.0.0.1:51096_solr-n_0000000000) [n:127.0.0.1:51096_solr    ] o.a.s.c.OverseerAutoReplicaFailoverThread Replica core_node2 may need to failover.
   [junit4]   2> 290041 WARN  (OverseerHdfsCoreFailoverThread-98726144763363334-127.0.0.1:51096_solr-n_0000000000) [n:127.0.0.1:51096_solr    ] o.a.s.c.OverseerAutoReplicaFailoverThread Replica core_node4 may need to failover.
   [junit4]   2> 290269 INFO  (zkCallback-181-thread-1-processing-n:127.0.0.1:51096_solr) [n:127.0.0.1:51096_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/backuprestore/state.json] for collection [backuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 290517 INFO  (qtp605417064-533) [n:127.0.0.1:34997_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=backuprestore&version=2&replicaType=NRT&property.customKey=customValue&coreNodeName=core_node13&name=backuprestore_shard2_replica_n10&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 290540 INFO  (qtp605417064-527) [n:127.0.0.1:34997_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=backuprestore&version=2&replicaType=PULL&property.customKey=customValue&coreNodeName=core_node9&name=backuprestore_shard1_replica_p6&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 290556 INFO  (qtp605417064-530) [n:127.0.0.1:34997_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=backuprestore&version=2&replicaType=PULL&property.customKey=customValue&coreNodeName=core_node16&name=backuprestore_shard2_replica_p14&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 290561 INFO  (zkCallback-181-thread-1-processing-n:127.0.0.1:51096_solr) [n:127.0.0.1:51096_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/backuprestore/state.json] for collection [backuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 290604 INFO  (zkCallback-180-thread-1-processing-n:127.0.0.1:34997_solr) [n:127.0.0.1:34997_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/backuprestore/state.json] for collection [backuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 290877 INFO  (qtp341142357-522) [n:127.0.0.1:51096_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=backuprestore&version=2&replicaType=NRT&property.customKey=customValue&coreNodeName=core_node2&name=backuprestore_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 290878 INFO  (qtp341142357-524) [n:127.0.0.1:51096_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=backuprestore&version=2&replicaType=TLOG&property.customKey=customValue&coreNodeName=core_node15&name=backuprestore_shard2_replica_t12&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 290881 INFO  (zkCallback-180-thread-2-processing-n:127.0.0.1:34997_solr) [n:127.0.0.1:34997_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/backuprestore/state.json] for collection [backuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 293935 INFO  (qtp605417064-528) [n:127.0.0.1:34997_solr c:backuprestore s:shard1 r:core_node4 x:backuprestore_shard1_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 294016 INFO  (qtp605417064-528) [n:127.0.0.1:34997_solr c:backuprestore s:shard1 r:core_node4 x:backuprestore_shard1_replica_n3] o.a.s.s.IndexSchema [backuprestore_shard1_replica_n3] Schema name=minimal
   [junit4]   2> 294039 INFO  (qtp605417064-528) [n:127.0.0.1:34997_solr c:backuprestore s:shard1 r:core_node4 x:backuprestore_shard1_replica_n3] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 294039 INFO  (qtp605417064-528) [n:127.0.0.1:34997_solr c:backuprestore s:shard1 r:core_node4 x:backuprestore_shard1_replica_n3] o.a.s.c.CoreContainer Creating SolrCore 'backuprestore_shard1_replica_n3' using configuration from collection backuprestore, trusted=true
   [junit4]   2> 294077 INFO  (qtp605417064-528) [n:127.0.0.1:34997_solr c:backuprestore s:shard1 r:core_node4 x:backuprestore_shard1_replica_n3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34997.solr.core.backuprestore.shard1.replica_n3' (registry 'solr.core.backuprestore.shard1.replica_n3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6db954c1
   [junit4]   2> 294077 INFO  (qtp605417064-528) [n:127.0.0.1:34997_solr c:backuprestore s:shard1 r:core_node4 x:backuprestore_shard1_replica_n3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 294077 INFO  (qtp605417064-528) [n:127.0.0.1:34997_solr c:backuprestore s:shard1 r:core_node4 x:backuprestore_shard1_replica_n3] o.a.s.c.SolrCore [[backuprestore_shard1_replica_n3] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.TestLocalFSCloudBackupRestore_A18CA5ECF70A4418-001/tempDir-001/node2/backuprestore_shard1_replica_n3], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.TestLocalFSCloudBackupRestore_A18CA5ECF70A4418-001/tempDir-001/node2/./backuprestore_shard1_replica_n3/data/]
   [junit4]   2> 294586 INFO  (zkCallback-180-thread-2-processing-n:127.0.0.1:34997_solr) [n:127.0.0.1:34997_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/backuprestore/state.json] for collection [backuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 294586 INFO  (zkCallback-180-thread-1-processing-n:127.0.0.1:34997_solr) [n:127.0.0.1:34997_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/backuprestore/state.json] for collection [backuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 294588 INFO  (zkCallback-181-thread-1-processing-n:127.0.0.1:51096_solr) [n:127.0.0.1:51096_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/backuprestore/state.json] for collection [backuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 294730 INFO  (qtp341142357-518) [n:127.0.0.1:51096_solr c:backuprestore s:shard2 r:core_node11 x:backuprestore_shard2_replica_n8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 294739 INFO  (qtp341142357-519) [n:127.0.0.1:51096_solr c:backuprestore s:shard1 r:core_node7 x:backuprestore_shard1_replica_t5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 294788 INFO  (qtp341142357-518) [n:127.0.0.1:51096_solr c:backuprestore s:shard2 r:core_node11 x:backuprestore_shard2_replica_n8] o.a.s.s.IndexSchema [backuprestore_shard2_replica_n8] Schema name=minimal
   [junit4]   2> 294845 INFO  (zkCallback-181-thread-1-processing-n:127.0.0.1:51096_solr) [n:127.0.0.1:51096_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/backuprestore/state.json] for collection [backuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 294847 INFO  (qtp341142357-519) [n:127.0.0.1:51096_solr c:backuprestore s:shard1 r:core_node7 x:backuprestore_shard1_replica_t5] o.a.s.s.IndexSchema [backuprestore_shard1_replica_t5] Schema name=minimal
   [junit4]   2> 294856 INFO  (zkCallback-180-thread-2-processing-n:127.0.0.1:34997_solr) [n:127.0.0.1:34997_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/backuprestore/state.json] for collection [backuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 294856 INFO  (zkCallback-180-thread-1-processing-n:127.0.0.1:34997_solr) [n:127.0.0.1:34997_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/backuprestore/state.json] for collection [backuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 294894 INFO  (qtp341142357-518) [n:127.0.0.1:51096_solr c:backuprestore s:shard2 r:core_node11 x:backuprestore_shard2_replica_n8] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 294894 INFO  (qtp341142357-518) [n:127.0.0.1:51096_solr c:backuprestore s:shard2 r:core_node11 x:backuprestore_shard2_replica_n8] o.a.s.c.CoreContainer Creating SolrCore 'backuprestore_shard2_replica_n8' using configuration from collection backuprestore, trusted=true
   [junit4]   2> 294895 INFO  (qtp341142357-518) [n:127.0.0.1:51096_solr c:backuprestore s:shard2 r:core_node11 x:backuprestore_shard2_replica_n8] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51096.solr.core.backuprestore.shard2.replica_n8' (registry 'solr.core.backuprestore.shard2.replica_n8') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6db954c1
   [junit4]   2> 294895 INFO  (qtp341142357-518) [n:127.0.0.1:51096_solr c:backuprestore s:shard2 r:core_node11 x:backuprestore_shard2_replica_n8] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 294895 INFO  (qtp341142357-518) [n:127.0.0.1:51096_solr c:backuprestore s:shard2 r:core_node11 x:backuprestore_shard2_replica_n8] o.a.s.c.SolrCore [[backuprestore_shard2_replica_n8] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.TestLocalFSCloudBackupRestore_A18CA5ECF70A4418-001/tempDir-001/node1/backuprestore_shard2_replica_n8], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.TestLocalFSCloudBackupRestore_A18CA5ECF70A4418-001/tempDir-001/node1/./backuprestore_shard2_replica_n8/data/]
   [junit4]   2> 294933 INFO  (qtp341142357-519) [n:127.0.0.1:51096_solr c:backuprestore s:shard1 r:core_node7 x:backuprestore_shard1_replica_t5] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 294933 INFO  (qtp341142357-519) [n:127.0.0.1:51096_solr c:backuprestore s:shard1 r:core_node7 x:backuprestore_shard1_replica_t5] o.a.s.c.CoreContainer Creating SolrCore 'backuprestore_shard1_replica_t5' using configuration from collection backuprestore, trusted=true
   [junit4]   2> 295016 INFO  (qtp341142357-519) [n:127.0.0.1:51096_solr c:backuprestore s:shard1 r:core_node7 x:backuprestore_shard1_replica_t5] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51096.solr.core.backuprestore.shard1.replica_t5' (registry 'solr.core.backuprestore.shard1.replica_t5') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6db954c1
   [junit4]   2> 295017 INFO  (qtp341142357-519) [n:127.0.0.1:51096_solr c:backuprestore s:shard1 r:core_node7 x:backuprestore_shard1_replica_t5] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 295017 INFO  (qtp341142357-519) [n:127.0.0.1:51096_solr c:backuprestore s:shard1 r:core_node7 x:backuprestore_shard1_replica_t5] o.a.s.c.SolrCore [[backuprestore_shard1_replica_t5] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/

[...truncated too long message...]

 c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=382944990
   [junit4]   2> 856073 INFO  (zkCallback-23-thread-19-processing-n:127.0.0.1:32965_solr) [n:127.0.0.1:32965_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 856073 INFO  (zkCallback-25-thread-12-processing-n:127.0.0.1:35232_solr) [n:127.0.0.1:35232_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 856098 INFO  (zkCallback-22-thread-15-processing-n:127.0.0.1:45822_solr) [n:127.0.0.1:45822_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 856664 INFO  (coreCloseExecutor-235-thread-1) [n:127.0.0.1:35232_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.SolrCore [collection1_shard2_replica_n2]  CLOSING SolrCore org.apache.solr.core.SolrCore@3cc6f8bf
   [junit4]   2> 856664 INFO  (coreCloseExecutor-235-thread-1) [n:127.0.0.1:35232_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1.shard2.replica_n2, tag=1019672767
   [junit4]   2> 856664 INFO  (coreCloseExecutor-235-thread-1) [n:127.0.0.1:35232_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@395001e: rootName = solr_35232, domain = solr.core.collection1.shard2.replica_n2, service url = null, agent id = null] for registry solr.core.collection1.shard2.replica_n2 / com.codahale.metrics.MetricRegistry@250aa59a
   [junit4]   2> 856676 INFO  (zkCallback-23-thread-19-processing-n:127.0.0.1:32965_solr) [n:127.0.0.1:32965_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 856732 INFO  (jetty-closer-4-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@615c64e3{/solr,null,UNAVAILABLE}
   [junit4]   2> 856735 INFO  (zkCallback-25-thread-11-processing-n:127.0.0.1:35232_solr) [n:127.0.0.1:35232_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 856751 INFO  (coreCloseExecutor-235-thread-1) [n:127.0.0.1:35232_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard2.leader, tag=1019672767
   [junit4]   2> 856820 WARN  (zkCallback-23-thread-19-processing-n:127.0.0.1:32965_solr) [n:127.0.0.1:32965_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 856834 INFO  (jetty-closer-4-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@21e7a34e{/solr,null,UNAVAILABLE}
   [junit4]   2> 856836 INFO  (zkCallback-25-thread-12-processing-n:127.0.0.1:35232_solr) [n:127.0.0.1:35232_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 856836 INFO  (zkCallback-25-thread-11-processing-n:127.0.0.1:35232_solr) [n:127.0.0.1:35232_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35232_solr
   [junit4]   2> 857575 WARN  (zkCallback-24-thread-15-processing-n:127.0.0.1:42702_solr) [n:127.0.0.1:42702_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 857575 INFO  (jetty-closer-4-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@711492e2{/solr,null,UNAVAILABLE}
   [junit4]   2> 858371 WARN  (zkCallback-25-thread-11-processing-n:127.0.0.1:35232_solr) [n:127.0.0.1:35232_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 858372 INFO  (jetty-closer-4-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6f714bc{/solr,null,UNAVAILABLE}
   [junit4]   2> 858380 ERROR (SUITE-StreamExpressionTest-seed#[A1497D5BF4154CA7]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 858381 INFO  (SUITE-StreamExpressionTest-seed#[A1497D5BF4154CA7]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53699 53699
   [junit4]   2> 858550 INFO  (Thread-13) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53699 53699
   [junit4]   2> 858551 WARN  (Thread-13) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 100 /solr/aliases.json
   [junit4]   2> 100 /solr/clusterprops.json
   [junit4]   2> 13 /solr/configs/conf
   [junit4]   2> 8 /solr/configs/ml
   [junit4]   2> 7 /solr/security.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 100 /solr/clusterstate.json
   [junit4]   2> 47 /solr/collections/destinationCollection/state.json
   [junit4]   2> 37 /solr/collections/parallelDestinationCollection1/state.json
   [junit4]   2> 32 /solr/collections/mainCorpus/state.json
   [junit4]   2> 30 /solr/collections/parallelDestinationCollection/state.json
   [junit4]   2> 28 /solr/collections/workQueue/state.json
   [junit4]   2> 25 /solr/collections/destination/state.json
   [junit4]   2> 16 /solr/collections/collection1/state.json
   [junit4]   2> 16 /solr/collections/uknownCollection/state.json
   [junit4]   2> 14 /solr/collections/checkpointCollection/state.json
   [junit4]   2> 12 /solr/collections/modelCollection/state.json
   [junit4]   2> 3 /solr/overseer_elect/election/98726507348754442-127.0.0.1:32965_solr-n_0000000000
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 100 /solr/collections
   [junit4]   2> 96 /solr/live_nodes
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_A1497D5BF4154CA7-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {name_s=PostingsFormat(name=Memory), expr_s=BlockTreeOrds(blocksize=128), terms_ss=Lucene50(blocksize=128), join2_s=Lucene50(blocksize=128), field_s=PostingsFormat(name=Direct), multiDefault=Lucene50(blocksize=128), subject=Lucene50(blocksize=128), a1_s=PostingsFormat(name=Direct), a2_s=BlockTreeOrds(blocksize=128), level1_s=BlockTreeOrds(blocksize=128), body_t=PostingsFormat(name=Memory), s_multi=BlockTreeOrds(blocksize=128), col_s=PostingsFormat(name=Direct), level2_s=PostingsFormat(name=Direct), a_ss=Lucene50(blocksize=128), a_s=BlockTreeOrds(blocksize=128), tv_text=PostingsFormat(name=Direct), term_s=Lucene50(blocksize=128), a_t=Lucene50(blocksize=128), text_s=BlockTreeOrds(blocksize=128), id=PostingsFormat(name=Memory), text=BlockTreeOrds(blocksize=128), featureSet_s=PostingsFormat(name=Direct), checkpoint_ss=PostingsFormat(name=Memory), test_t=PostingsFormat(name=Memory), ident_s=PostingsFormat(name=Direct), whitetok=PostingsFormat(name=Memory), side_s=PostingsFormat(name=Memory)}, docValues:{name_s=DocValuesFormat(name=Lucene70), expr_s=DocValuesFormat(name=Memory), iteration_i=DocValuesFormat(name=Lucene70), terms_ss=DocValuesFormat(name=Direct), join2_s=DocValuesFormat(name=Direct), multiDefault=DocValuesFormat(name=Direct), intDefault=DocValuesFormat(name=Asserting), a_f=DocValuesFormat(name=Asserting), falseNegative_i=DocValuesFormat(name=Asserting), a2_s=DocValuesFormat(name=Memory), a_i=DocValuesFormat(name=Lucene70), level1_s=DocValuesFormat(name=Memory), a_s=DocValuesFormat(name=Memory), id=DocValuesFormat(name=Lucene70), alpha_d=DocValuesFormat(name=Lucene70), i_multi=DocValuesFormat(name=Lucene70), idf_d=DocValuesFormat(name=Asserting), b_ls=DocValuesFormat(name=Asserting), checkpoint_ss=DocValuesFormat(name=Lucene70), ident_s=DocValuesFormat(name=Asserting), order_i=DocValuesFormat(name=Direct), error_d=DocValuesFormat(name=Memory), side_s=DocValuesFormat(name=Lucene70), truePositive_i=DocValuesFormat(name=Memory), miles_i=DocValuesFormat(name=Asserting), field_s=DocValuesFormat(name=Asserting), price_f=DocValuesFormat(name=Direct), a1_s=DocValuesFormat(name=Asserting), join1_i=DocValuesFormat(name=Memory), join3_i=DocValuesFormat(name=Lucene70), test_i=DocValuesFormat(name=Direct), falsePositive_i=DocValuesFormat(name=Asserting), field_i=DocValuesFormat(name=Direct), s_multi=DocValuesFormat(name=Memory), col_s=DocValuesFormat(name=Asserting), level2_s=DocValuesFormat(name=Asserting), a_ss=DocValuesFormat(name=Direct), score_f=DocValuesFormat(name=Memory), term_s=DocValuesFormat(name=Direct), text_s=DocValuesFormat(name=Memory), timestamp=DocValuesFormat(name=Direct), idfs_ds=DocValuesFormat(name=Direct), featureSet_s=DocValuesFormat(name=Asserting), weights_ds=DocValuesFormat(name=Lucene70), trueNegative_i=DocValuesFormat(name=Memory), index_i=DocValuesFormat(name=Asserting), test_dt=DocValuesFormat(name=Memory), out_i=DocValuesFormat(name=Asserting), _version_=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1626, maxMBSortInHeap=6.591176188128704, sim=RandomSimilarity(queryNorm=true): {}, locale=sr-CS, timezone=America/St_Thomas
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=179963392,total=395837440
   [junit4]   2> NOTE: All tests run in this JVM: [ReverseEvaluatorTest, SolrExampleJettyTest, StreamExpressionTest]
   [junit4] Completed [122/145 (2!)] on J2 in 725.12s, 113 tests, 1 error <<< FAILURES!

[...truncated 45754 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-master - Build # 2106 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/2106/

7 tests failed.
FAILED:  org.apache.lucene.classification.CachingNaiveBayesClassifierTest.testPerformance

Error Message:
evaluation took more than 1m: 72s

Stack Trace:
java.lang.AssertionError: evaluation took more than 1m: 72s
        at __randomizedtesting.SeedInfo.seed([48DBB807A000B204:8F3A4A25CBB48AAB]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.lucene.classification.CachingNaiveBayesClassifierTest.testPerformance(CachingNaiveBayesClassifierTest.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.addReplicaTest

Error Message:
Could not load collection from ZK: addReplicaColl

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: addReplicaColl
        at __randomizedtesting.SeedInfo.seed([C48A3674D8CF3C7D:572A615256C777AC]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1115)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:128)
        at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:108)
        at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:247)
        at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.addReplicaTest(CollectionsAPIDistributedZkTest.java:651)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/addReplicaColl/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1127)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1113)
        ... 44 more


FAILED:  org.apache.solr.cloud.TestTlogReplica.testCreateDelete {seed=[C48A3674D8CF3C7D:6BCC7BC41AFD0F61]}

Error Message:
Error from server at https://127.0.0.1:45447/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000016

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:45447/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000016
        at __randomizedtesting.SeedInfo.seed([C48A3674D8CF3C7D:6BCC7BC41AFD0F61]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.TestTlogReplica.testCreateDelete(TestTlogReplica.java:211)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.MultiThreadedOCPTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([C48A3674D8CF3C7D:4CDE09AE76335185]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.MultiThreadedOCPTest.testFillWorkQueue(MultiThreadedOCPTest.java:112)
        at org.apache.solr.cloud.MultiThreadedOCPTest.test(MultiThreadedOCPTest.java:67)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:


Stack Trace:
java.lang.NullPointerException
        at __randomizedtesting.SeedInfo.seed([C48A3674D8CF3C7D:57A4FD8F59FF6DA]:0)
        at org.apache.solr.cloud.TestTlogReplica.getSolrRunner(TestTlogReplica.java:874)
        at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:517)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testOnlyLeaderIndexes

Error Message:


Stack Trace:
java.lang.NullPointerException
        at __randomizedtesting.SeedInfo.seed([C48A3674D8CF3C7D:D88B4BF9AD6A42EE]:0)
        at org.apache.solr.cloud.TestTlogReplica.getSolrCore(TestTlogReplica.java:839)
        at org.apache.solr.cloud.TestTlogReplica.testOnlyLeaderIndexes(TestTlogReplica.java:466)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testOutOfOrderDBQWithInPlaceUpdates

Error Message:


Stack Trace:
java.lang.NullPointerException
        at __randomizedtesting.SeedInfo.seed([C48A3674D8CF3C7D:424BCE99879EEA9D]:0)
        at org.apache.solr.cloud.TestTlogReplica.getSolrCore(TestTlogReplica.java:839)
        at org.apache.solr.cloud.TestTlogReplica.testOutOfOrderDBQWithInPlaceUpdates(TestTlogReplica.java:645)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 6071 lines...]
   [junit4] Suite: org.apache.lucene.classification.CachingNaiveBayesClassifierTest
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CachingNaiveBayesClassifierTest -Dtests.method=testPerformance -Dtests.seed=48DBB807A000B204 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=de-CH -Dtests.timezone=Asia/Kuala_Lumpur -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 87.2s J2 | CachingNaiveBayesClassifierTest.testPerformance <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: evaluation took more than 1m: 72s
   [junit4]    > at __randomizedtesting.SeedInfo.seed([48DBB807A000B204:8F3A4A25CBB48AAB]:0)
   [junit4]    > at org.apache.lucene.classification.CachingNaiveBayesClassifierTest.testPerformance(CachingNaiveBayesClassifierTest.java:111)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {bool=IB SPL-DZ(0.3), cat=DFI(ChiSquared), text=IB LL-L2}, locale=de-CH, timezone=Asia/Kuala_Lumpur
   [junit4]   2> NOTE: Linux 4.4.0-83-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=272234304,total=333971456
   [junit4]   2> NOTE: All tests run in this JVM: [CachingNaiveBayesClassifierTest]
   [junit4] Completed [5/11 (1!)] on J2 in 100.01s, 4 tests, 1 failure <<< FAILURES!

[...truncated 6638 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> 4092586 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[C48A3674D8CF3C7D]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C48A3674D8CF3C7D-001/init-core-data-001
   [junit4]   2> 4092587 WARN  (SUITE-CollectionsAPIDistributedZkTest-seed#[C48A3674D8CF3C7D]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=66 numCloses=66
   [junit4]   2> 4092587 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[C48A3674D8CF3C7D]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 4092588 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[C48A3674D8CF3C7D]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 4092623 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[C48A3674D8CF3C7D]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C48A3674D8CF3C7D-001/tempDir-001
   [junit4]   2> 4092623 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[C48A3674D8CF3C7D]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 4092623 INFO  (Thread-2325) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 4092623 INFO  (Thread-2325) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 4092643 ERROR (Thread-2325) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 4092732 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[C48A3674D8CF3C7D]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:43173
   [junit4]   2> 4092867 INFO  (jetty-launcher-1430-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 4092886 INFO  (jetty-launcher-1430-thread-3) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 4092903 INFO  (jetty-launcher-1430-thread-4) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 4092906 INFO  (jetty-launcher-1430-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 4093016 INFO  (jetty-launcher-1430-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2c63f38b{/solr,null,AVAILABLE}
   [junit4]   2> 4093017 INFO  (jetty-launcher-1430-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@646c3be9{SSL,[ssl, http/1.1]}{127.0.0.1:36152}
   [junit4]   2> 4093017 INFO  (jetty-launcher-1430-thread-2) [    ] o.e.j.s.Server Started @4116804ms
   [junit4]   2> 4093017 INFO  (jetty-launcher-1430-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36152}
   [junit4]   2> 4093017 ERROR (jetty-launcher-1430-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4093017 INFO  (jetty-launcher-1430-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 4093017 INFO  (jetty-launcher-1430-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4093017 INFO  (jetty-launcher-1430-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 4093017 INFO  (jetty-launcher-1430-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-27T20:43:10.064Z
   [junit4]   2> 4093110 INFO  (jetty-launcher-1430-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@c931c1a{/solr,null,AVAILABLE}
   [junit4]   2> 4093110 INFO  (jetty-launcher-1430-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@662b426a{SSL,[ssl, http/1.1]}{127.0.0.1:40197}
   [junit4]   2> 4093110 INFO  (jetty-launcher-1430-thread-3) [    ] o.e.j.s.Server Started @4116898ms
   [junit4]   2> 4093110 INFO  (jetty-launcher-1430-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40197}
   [junit4]   2> 4093110 ERROR (jetty-launcher-1430-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4093110 INFO  (jetty-launcher-1430-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 4093111 INFO  (jetty-launcher-1430-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4093111 INFO  (jetty-launcher-1430-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 4093111 INFO  (jetty-launcher-1430-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-27T20:43:10.158Z
   [junit4]   2> 4093142 INFO  (jetty-launcher-1430-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 4093252 INFO  (jetty-launcher-1430-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@775c60{/solr,null,AVAILABLE}
   [junit4]   2> 4093253 INFO  (jetty-launcher-1430-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3742cb19{SSL,[ssl, http/1.1]}{127.0.0.1:43811}
   [junit4]   2> 4093253 INFO  (jetty-launcher-1430-thread-1) [    ] o.e.j.s.Server Started @4117040ms
   [junit4]   2> 4093253 INFO  (jetty-launcher-1430-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43811}
   [junit4]   2> 4093253 ERROR (jetty-launcher-1430-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4093253 INFO  (jetty-launcher-1430-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 4093253 INFO  (jetty-launcher-1430-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4093253 INFO  (jetty-launcher-1430-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 4093253 INFO  (jetty-launcher-1430-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-27T20:43:10.300Z
   [junit4]   2> 4093309 INFO  (jetty-launcher-1430-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@66420119{/solr,null,AVAILABLE}
   [junit4]   2> 4093309 INFO  (jetty-launcher-1430-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@66ce9de2{SSL,[ssl, http/1.1]}{127.0.0.1:37627}
   [junit4]   2> 4093309 INFO  (jetty-launcher-1430-thread-4) [    ] o.e.j.s.Server Started @4117097ms
   [junit4]   2> 4093309 INFO  (jetty-launcher-1430-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37627}
   [junit4]   2> 4093309 ERROR (jetty-launcher-1430-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4093309 INFO  (jetty-launcher-1430-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 4093309 INFO  (jetty-launcher-1430-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4093309 INFO  (jetty-launcher-1430-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 4093310 INFO  (jetty-launcher-1430-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-27T20:43:10.356Z
   [junit4]   2> 4093341 INFO  (jetty-launcher-1430-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 4093399 INFO  (jetty-launcher-1430-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43173/solr
   [junit4]   2> 4093399 INFO  (jetty-launcher-1430-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 4093400 INFO  (jetty-launcher-1430-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 4093689 INFO  (jetty-launcher-1430-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43173/solr
   [junit4]   2> 4093723 INFO  (jetty-launcher-1430-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43173/solr
   [junit4]   2> 4093797 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ec51435f90007, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 4093853 INFO  (jetty-launcher-1430-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43173/solr
   [junit4]   2> 4094012 INFO  (jetty-launcher-1430-thread-2) [n:127.0.0.1:36152_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 4094012 INFO  (jetty-launcher-1430-thread-2) [n:127.0.0.1:36152_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36152_solr
   [junit4]   2> 4094013 INFO  (jetty-launcher-1430-thread-2) [n:127.0.0.1:36152_solr    ] o.a.s.c.Overseer Overseer (id=98732932444258314-127.0.0.1:36152_solr-n_0000000000) starting
   [junit4]   2> 4094304 INFO  (jetty-launcher-1430-thread-2) [n:127.0.0.1:36152_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36152_solr
   [junit4]   2> 4094401 INFO  (zkCallback-1451-thread-1-processing-n:127.0.0.1:36152_solr) [n:127.0.0.1:36152_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4094657 INFO  (jetty-launcher-1430-thread-1) [n:127.0.0.1:43811_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4094706 INFO  (jetty-launcher-1430-thread-3) [n:127.0.0.1:40197_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4094840 INFO  (jetty-launcher-1430-thread-1) [n:127.0.0.1:43811_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 4094889 INFO  (jetty-launcher-1430-thread-3) [n:127.0.0.1:40197_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 4094967 INFO  (jetty-launcher-1430-thread-1) [n:127.0.0.1:43811_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43811_solr
   [junit4]   2> 4095003 INFO  (zkCallback-1451-thread-1-processing-n:127.0.0.1:36152_solr) [n:127.0.0.1:36152_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4095025 INFO  (jetty-launcher-1430-thread-2) [n:127.0.0.1:36152_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5634d705
   [junit4]   2> 4095027 INFO  (zkCallback-1447-thread-1-processing-n:127.0.0.1:43811_solr) [n:127.0.0.1:43811_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4095027 INFO  (zkCallback-1452-thread-1-processing-n:127.0.0.1:40197_solr) [n:127.0.0.1:40197_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4095073 INFO  (jetty-launcher-1430-thread-3) [n:127.0.0.1:40197_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40197_solr
   [junit4]   2> 4095133 INFO  (zkCallback-1451-thread-1-processing-n:127.0.0.1:36152_solr) [n:127.0.0.1:36152_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4095133 INFO  (zkCallback-1447-thread-1-processing-n:127.0.0.1:43811_solr) [n:127.0.0.1:43811_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4095135 INFO  (jetty-launcher-1430-thread-2) [n:127.0.0.1:36152_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5634d705
   [junit4]   2> 4095135 INFO  (jetty-launcher-1430-thread-2) [n:127.0.0.1:36152_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5634d705
   [junit4]   2> 4095136 INFO  (jetty-launcher-1430-thread-2) [n:127.0.0.1:36152_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C48A3674D8CF3C7D-001/tempDir-001/node1
   [junit4]   2> 4095166 INFO  (zkCallback-1452-thread-1-processing-n:127.0.0.1:40197_solr) [n:127.0.0.1:40197_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4095756 INFO  (jetty-launcher-1430-thread-3) [n:127.0.0.1:40197_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5634d705
   [junit4]   2> 4095828 INFO  (jetty-launcher-1430-thread-4) [n:127.0.0.1:37627_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 4095883 INFO  (jetty-launcher-1430-thread-3) [n:127.0.0.1:40197_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5634d705
   [junit4]   2> 4095883 INFO  (jetty-launcher-1430-thread-3) [n:127.0.0.1:40197_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5634d705
   [junit4]   2> 4095884 INFO  (jetty-launcher-1430-thread-3) [n:127.0.0.1:40197_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C48A3674D8CF3C7D-001/tempDir-001/node2
   [junit4]   2> 4095884 INFO  (jetty-launcher-1430-thread-1) [n:127.0.0.1:43811_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5634d705
   [junit4]   2> 4095898 INFO  (jetty-launcher-1430-thread-4) [n:127.0.0.1:37627_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 4095901 INFO  (jetty-launcher-1430-thread-4) [n:127.0.0.1:37627_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37627_solr
   [junit4]   2> 4095978 INFO  (jetty-launcher-1430-thread-1) [n:127.0.0.1:43811_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5634d705
   [junit4]   2> 4095978 INFO  (jetty-launcher-1430-thread-1) [n:127.0.0.1:43811_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5634d705
   [junit4]   2> 4096007 INFO  (zkCallback-1452-thread-1-processing-n:127.0.0.1:40197_solr) [n:127.0.0.1:40197_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4096007 INFO  (zkCallback-1447-thread-1-processing-n:127.0.0.1:43811_solr) [n:127.0.0.1:43811_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4096084 INFO  (jetty-launcher-1430-thread-1) [n:127.0.0.1:43811_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C48A3674D8CF3C7D-001/tempDir-001/node3
   [junit4]   2> 4096085 INFO  (zkCallback-1451-thread-2-processing-n:127.0.0.1:36152_solr) [n:127.0.0.1:36152_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4096226 INFO  (zkCallback-1446-thread-1-processing-n:127.0.0.1:37627_solr) [n:127.0.0.1:37627_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4097082 INFO  (jetty-launcher-1430-thread-4) [n:127.0.0.1:37627_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5634d705
   [junit4]   2> 4097227 INFO  (jetty-launcher-1430-thread-4) [n:127.0.0.1:37627_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5634d705
   [junit4]   2> 4097227 INFO  (jetty-launcher-1430-thread-4) [n:127.0.0.1:37627_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5634d705
   [junit4]   2> 4097228 INFO  (jetty-launcher-1430-thread-4) [n:127.0.0.1:37627_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C48A3674D8CF3C7D-001/tempDir-001/node4
   [junit4]   2> 4097979 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[C48A3674D8CF3C7D]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 4097980 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[C48A3674D8CF3C7D]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43173/solr ready
   [junit4]   2> 4098164 INFO  (TEST-CollectionsAPIDistributedZkTest.testMissingRequiredParameters-seed#[C48A3674D8CF3C7D]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testMissingRequiredParameters
   [junit4]   2> 4098177 INFO  (TEST-CollectionsAPIDistributedZkTest.testMissingRequiredParameters-seed#[C48A3674D8CF3C7D]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 4098508 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4098509 ERROR (qtp366016851-9338) [n:127.0.0.1:43811_solr    ] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Missing required parameter: name
   [junit4]   2> at org.apache.solr.common.params.RequiredSolrParams.getParams(RequiredSolrParams.java:90)
   [junit4]   2> at org.apache.solr.common.params.SolrParams.getAll(SolrParams.java:462)
   [junit4]   2> at org.apache.solr.common.params.SolrParams.getAll(SolrParams.java:479)
   [junit4]   2> at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$0(CollectionsHandler.java:391)
   [junit4]   2> at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:936)
   [junit4]   2> at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:223)
   [junit4]   2> at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:210)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 4098545 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CREATE&numShards=2&wt=javabin&version=2} status=400 QTime=36
   [junit4]   2> 4098545 INFO  (TEST-CollectionsAPIDistributedZkTest.testMissingRequiredParameters-seed#[C48A3674D8CF3C7D]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testMissingRequiredParameters
   [junit4]   2> 4098763 INFO  (TEST-CollectionsAPIDistributedZkTest.testTooManyReplicas-seed#[C48A3674D8CF3C7D]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testTooManyReplicas
   [junit4]   2> 4098763 INFO  (TEST-CollectionsAPIDistributedZkTest.testTooManyReplicas-seed#[C48A3674D8CF3C7D]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 4098765 INFO  (qtp366016851-9339) [n:127.0.0.1:43811_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=10&collection.configName=conf&name=collection&nrtReplicas=10&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4098819 INFO  (OverseerThreadFactory-3719-thread-1-processing-n:127.0.0.1:36152_solr) [n:127.0.0.1:36152_solr    ] o.a.s.c.CreateCollectionCmd Create collection collection
   [junit4]   2> 4098819 WARN  (OverseerThreadFactory-3719-thread-1-processing-n:127.0.0.1:36152_solr) [n:127.0.0.1:36152_solr    ] o.a.s.c.CreateCollectionCmd Specified number of replicas of 10 on collection collection is higher than the number of Solr instances currently live or live and part of your createNodeSet(4). It's unusual to run two replica of the same slice on the same Solr-instance.
   [junit4]   2> 4098820 ERROR (OverseerThreadFactory-3719-thread-1-processing-n:127.0.0.1:36152_solr) [n:127.0.0.1:36152_solr    ] o.a.s.c.OverseerCollectionMessageHandler Collection: collection operation: create failed:org.apache.solr.common.SolrException: Cannot create collection collection. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 4. This allows a maximum of 4 to be created. Value of numShards is 2, value of nrtReplicas is 10, value of tlogReplicas is 0 and value of pullReplicas is 0. This requires 20 shards to be created (higher than the allowed number)
   [junit4]   2> at org.apache.solr.cloud.CreateCollectionCmd.call(CreateCollectionCmd.java:164)
   [junit4]   2> at org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:228)
   [junit4]   2> at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:464)
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 4098905 INFO  (qtp366016851-9339) [n:127.0.0.1:43811_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=10&collection.configName=conf&name=collection&nrtReplicas=10&action=CREATE&numShards=2&wt=javabin&version=2} status=400 QTime=139
   [junit4]   2> 4098908 INFO  (TEST-CollectionsAPIDistributedZkTest.testTooManyReplicas-seed#[C48A3674D8CF3C7D]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testTooManyReplicas
   [junit4]   2> 4099221 INFO  (TEST-CollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[C48A3674D8CF3C7D]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCreationAndDeletion
   [junit4]   2> 4099223 INFO  (TEST-CollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[C48A3674D8CF3C7D]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 4099224 INFO  (qtp366016851-9340) [n:127.0.0.1:43811_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf&name=created_and_deleted&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4099278 INFO  (OverseerThreadFactory-3719-thread-2-processing-n:127.0.0.1:36152_solr) [n:127.0.0.1:36152_solr    ] o.a.s.c.CreateCollectionCmd Create collection created_and_deleted
   [junit4]   2> 4099278 INFO  (OverseerCollectionConfigSetProcessor-98732932444258314-127.0.0.1:36152_solr-n_0000000000) [n:127.0.0.1:36152_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 4099385 INFO  (OverseerStateUpdate-98732932444258314-127.0.0.1:36152_solr-n_0000000000) [n:127.0.0.1:36152_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"created_and_deleted",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"created_and_deleted_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:43811/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 4099697 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=created_and_deleted_shard1_replica_n1&action=CREATE&numShards=1&collection=created_and_deleted&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 4099698 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 4099835 INFO  (zkCallback-1447-thread-1-processing-n:127.0.0.1:43811_solr) [n:127.0.0.1:43811_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4099844 INFO  (zkCallback-1447-thread-2-processing-n:127.0.0.1:43811_solr) [n:127.0.0.1:43811_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4100756 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 4100828 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.s.IndexSchema [created_and_deleted_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 4100901 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 4100901 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'created_and_deleted_shard1_replica_n1' using configuration from collection created_and_deleted, trusted=true
   [junit4]   2> 4100901 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.created_and_deleted.shard1.replica_n1' (registry 'solr.core.created_and_deleted.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5634d705
   [junit4]   2> 4100901 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 4100902 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.SolrCore [[created_and_deleted_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C48A3674D8CF3C7D-001/tempDir-001/node3/created_and_deleted_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C48A3674D8CF3C7D-001/tempDir-001/node3/created_and_deleted_shard1_replica_n1/data/]
   [junit4]   2> 4101613 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 4101613 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4101614 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4101614 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4101684 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@454ec275[created_and_deleted_shard1_replica_n1] main]
   [junit4]   2> 4101685 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 4101685 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 4101686 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 4101686 INFO  (searcherExecutor-3728-thread-1-processing-n:127.0.0.1:43811_solr x:created_and_deleted_shard1_replica_n1 s:shard1 c:created_and_deleted r:core_node2) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.SolrCore [created_and_deleted_shard1_replica_n1] Registered new searcher Searcher@454ec275[created_and_deleted_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4101686 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579726928591454208
   [junit4]   2> 4101757 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 4101757 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 4101757 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:43811/solr/created_and_deleted_shard1_replica_n1/
   [junit4]   2> 4101757 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 4101757 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:43811/solr/created_and_deleted_shard1_replica_n1/ has no replicas
   [junit4]   2> 4101757 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 4101759 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:43811/solr/created_and_deleted_shard1_replica_n1/ shard1
   [junit4]   2> 4101867 INFO  (zkCallback-1447-thread-2-processing-n:127.0.0.1:43811_solr) [n:127.0.0.1:43811_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4101867 INFO  (zkCallback-1447-thread-1-processing-n:127.0.0.1:43811_solr) [n:127.0.0.1:43811_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4101924 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 4101925 INFO  (qtp366016851-9338) [n:127.0.0.1:43811_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=created_and_deleted_shard1_replica_n1&action=CREATE&numShards=1&collection=created_and_deleted&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2227
   [junit4]   2> 4101969 INFO  (qtp366016851-9340) [n:127.0.0.1:43811_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 4102049 INFO  (zkCallback-1447-thread-2-processing-n:127.0.0.1:43811_solr) [n:127.0.0.1:43811_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4102049 INFO  (zkCallback-1447-thread-1-processing-n:127.0.0.1:43811_solr) [n:127.0.0.1:43811_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4102984 INFO  (qtp366016851-9340) [n:127.0.0.1:43811_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf&name=created_and_deleted&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=3760
   [junit4]   2> 4103001 INFO  (qtp366016851-9333) [n:127.0.0.1:43811_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4103001 INFO  (qtp366016851-9333) [n:127.0.0.1:43811_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 4103001 INFO  (qtp366016851-9336) [n:127.0.0.1:43811_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=created_and_deleted&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4103016 INFO  (OverseerCollectionConfigSetProcessor-98732932444258314-127.0.0.1:36152_solr-n_0000000000) [n:127.0.0.1:36152_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 4103017 INFO  (OverseerThreadFactory-3719-thread-3-processing-n:127.0.0.1:36152_solr) [n:127.0.0.1:36152_solr    ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 4103018 INFO  (qtp366016851-9339) [n:127.0.0.1:43811_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.created_and_deleted.shard1.replica_n1, tag=null
   [junit4]   2> 4103018 INFO  (qtp366016851-9339) [n:127.0.0.1:43811_solr    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@33314847: rootName = null, domain = solr.core.created_and_deleted.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.created_and_deleted.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@499842d4
   [junit4]   2> 4103167 INFO  (qtp366016851-9339) [n:127.0.0.1:43811_solr    ] o.a.s.c.SolrCore [created_and_deleted_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1da31f5c
   [junit4]   2> 4103168 INFO  (qtp366016851-9339) [n:127.0.0.1:43811_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.created_and_deleted.shard1.replica_n1, tag=497229660
   [junit4]   2> 4103168 INFO  (qtp366016851-9339) [n:127.0.0.1:43811_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.created_and_deleted.shard1.leader, tag=497229660
   [junit4]   2> 4103185 INFO  (qtp366016851-9339) [n:127.0.0.1:43811_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=created_and_deleted_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=167
   [junit4]   2> 4104004 INFO  (qtp366016851-9336) [n:127.0.0.1:43811_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=created_and_deleted&action=DELETE&wt=javabin&version=2} status=0 QTime=1002
   [junit4]   2> 4104005 INFO  (qtp366016851-9334) [n:127.0.0.1:43811_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4104005 INFO  (qtp366016851-9334) [n:127.0.0.1:43811_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 4104009 INFO  (TEST-CollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[C48A3674D8CF3C7D]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCreationAndDeletion
   [junit4]   2> 4104355 INFO  (TEST-CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes-seed#[C48A3674D8CF3C7D]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCoresAreDistributedAcrossNodes
   [junit4]   2> 4104355 INFO  (TEST-CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes-seed#[C48A3674D8CF3C7D]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 4104356 INFO  (qtp366016851-9334) [n:127.0.0.1:43811_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&name=nodes_used_collection&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4104414 INFO  (OverseerThreadFactory-3719-thread-4-processing-n:127.0.0.1:36152_solr) [n:127.0.0.1:36152_solr    ] o.a.s.c.CreateCollectionCmd Create collection nodes_used_collection
   [junit4]   2> 4104450 INFO  (OverseerCollectionConfigSetProcessor-98732932444258314-127.0.0.1:36152_solr-n_0000000000) [n:127.0.0.1:36152_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 4104758 INFO  (OverseerStateUpdate-98732932444258314-127.0.0.1:36152_solr-n_0000000000) [n:127.0.0.1:36152_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"nodes_used_collection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"nodes_used_collection_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:36152/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 4104760 INFO  (OverseerStateUpdate-98732932444258314-127.0.0.1:36152_solr-n_0000000000) [n:127.0.0.1:36152_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"nodes_used_collection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"nodes_used_collection_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:37627/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 4104810 INFO  (OverseerStateUpdate-98732932444258314-127.0.0.1:36152_solr-n_0000000000) [n:127.0.0.1:36152_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"nodes_used_collection",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"nodes_used_collection_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:43811/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 4104812 INFO  (OverseerStateUpdate-98732932444258314-127.0.0.1:36152_solr-n_0000000000) [n:127.0.0.1:36152_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"nodes_used_collection",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"nodes_used_collection_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:40197/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 4105049 INFO  (qtp366016851-9337) [n:127.0.0.1:43811_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard2_replica_n4&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 4105192 INFO  (zkCallback-1447-thread-1-processing-n:127.0.0.1:43811_solr) [n:127.0.0.1:43811_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4105194 INFO  (zkCallback-1447-thread-3-processing-n:127.0.0.1:43811_solr) [n:127.0.0.1:43811_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4106143 INFO  (qtp573619098-9349) [n:127.0.0.1:37627_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard1_replica_n2&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 4106144 INFO  (qtp573619098-9349) [n:127.0.0.1:37627_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 4106251 INFO  (qtp212087551-9315) [n:127.0.0.1:36152_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard1_replica_n1&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 4106252 INFO  (qtp212087551-9315) [n:127.0.0.1:36152_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 4106287 INFO  (qtp65763992-9326) [n:127.0.0.1:40197_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard2_replica_n6&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 4106287 INFO  (qtp65763992-9326) [n:127.0.0.1:40197_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 4106339 INFO  (qtp366016851-9337) [n:127.0.0.1:43811_solr c:nodes_used_collection s:shard2 r:core_node7 x:nodes_used_collection_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 4106434 INFO  (qtp366016851-9337) [n:127.0.0.1:43811_solr c:nodes_used_collection s:shard2 r:core_node7 x:nodes_used_collection_shard2_replica_n4] o.a.s.s.IndexSchema [nodes_used_collection_shard2_replica_n4] Schema name=minimal
   [junit4]   2> 4106436 INFO  (qtp366016851-9337) [n:127.0.0.1:43811_solr c:nodes_used_collection s:shard2 r:core_node7 x:nodes_used_collection_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 4106436 INFO  (qtp366016851-9337) [n:127.0.0.1:43811_solr c:nodes_used_collection s:shard2 r:core_node7 x:nodes_used_collection_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'nodes_used_collection_shard2_replica_n4' using configuration from collection nodes_used_collection, trusted=true
   [junit4]   2> 4106437 INFO  (zkCallback-1452-thread-1-processing-n:127.0.0.1:40197_solr) [n:127.0.0.1:40197_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4106437 INFO  (zkCallback-1451-thread-2-processing-n:127.0.0.1:36152_solr) [n:127.0.0.1:36152_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4106437 INFO  (zkCallback-1451-thread-1-processing-n:127.0.0.1:36152_solr) [n:127.0.0.1:36152_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4106449 INFO  (zkCallback-1447-thread-3-processing-n:127.0.0.1:43811_solr) [n:127.0.0.1:43811_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4106449 INFO  (zkCallback-1446-thread-1-processing-n:127.0.0.1:37627_solr) [n:127.0.0.1:37627_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4106449 INFO  (zkCallback-1447-thread-1-processing-n:127.0.0.1:43811_solr) [n:127.0.0.1:43811_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4106469 INFO  (zkCallback-1452-thread-2-processing-n:127.0.0.1:40197_solr) [n:127.0.0.1:40197_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4106470 INFO  (zkCallback-1446-thread-2-processing-n:127.0.0.1:37627_solr) [n:127.0.0.1:37627_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4106470 INFO  (qtp366016851-9337) [n:127.0.0.1:43811_solr c:nodes_used_collection s:shard2 r:core_node7 x:nodes_used_collection_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.nodes_used_collection.shard2.replica_n4' (registry 'solr.core.nodes_used_collection.shard2.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5634d705
   [junit4]   2> 4106470 INFO  (qtp366016851-9337) [n:127.0.0.1:43811_solr c:nodes_used_collection s:shard2 r:core_node7 x:nodes_used_collection_shard2_replica_n4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 4106471 INFO  (qtp366016851-9337) [n:127.0.0.1:43811_solr c:nodes_used_collection s:shard2 r:core_node7 x:nodes_used_collection_shard2_replica_n4] o.a.s.c.SolrCore [[nodes_used_collection_shard2_replica_n4] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C48A3674D8CF3C7D-001/tempDir-001/node3/nodes_used_collection_shard2_replica_n4], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C48A3674D8CF3C7D-001/tempDir-001/node3/nodes_used_collection_shard2_replica_n4/data/]
   [junit4]   2> 4107072 INFO  (qtp366016851-9337) [n:127.0.0.1:43811_solr c:nodes_used_collection s:shard2 r:core_node7 x:nodes_used_collection_shard2_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 4107072 INFO  (qtp366016851-9337) [n:127.0.0.1:43811_solr c:nodes_used_collection s:shard2 r:core_node7 x:nodes_used_collection_shard2_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4107073 INFO  (qtp366016851-9337) [n:127.0.0.1:43811_solr c:nodes_used_collection s:shard2 r:core_node7 x:nodes_used_collection_shard2_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4107073 INFO  (qtp366016851-9337) [n:127.0.0.1:43811_solr c:nodes_used_collection s:shard2 r:core_node7 x:nodes_used_collection_shard2_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4107074 INFO  (qtp366016851-9337) [n:127.0.0.1:43811_solr c:nodes_used_collection s:shard2 r:core_node7 x:nodes_used_collection_shard2_replica_n4] o.a.s.s.SolrIndexSearcher Opening [Searcher@512fc60[nodes_used_collection_shard2_replica_n4] main]
   [junit4]   2> 4107128 INFO  (qtp366016851-9337) [n:127.0.0.1:43811_solr c:nodes_used_collection s:shard2 r:core_node7 x:nodes_used_collection_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 4107128 INFO  (qtp366016851-9337) [n:127.0.0.1:43811_solr c:nodes_used_collection s:shard2 r:core_node7 x:nodes_used_collection_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 4107129 INFO  (qtp366016851-9337) [n:127.0.0.1:43811_solr c:nodes_used_collection s:shard2 r:core_node7 x:nodes_used_collection_shard2_replica_n4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 4107129 INFO  (searcherExecutor-3733-thread-1-processing-n:127.0.0.1:43811_solr x:nodes_used_collection_shard2_replica_n4 s:shard2 c:nodes_used_collection r:core_node7) [n:127.0.0.1:43811_solr c:nodes_used_collection s:shard2 r:core_node7 x:nodes_used_collection_shard2_replica_n4] o.a.s.c.SolrCore [nodes_used_collection_shard2_replica_n4] Registered new searcher Searcher@512fc60[nodes_used_collection_shard2_replica_n4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4107129 INFO  (qtp366016851-9337) [n:127.0.0.1:43811_solr c:nodes_used_collecti

[...truncated too long message...]

delete.shard1.replica_t4 / com.codahale.metrics.MetricRegistry@1f4d1be5
   [junit4]   2> 7175456 INFO  (coreCloseExecutor-6999-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.tlog_replica_test_create_delete.shard1.leader, tag=974239541
   [junit4]   2> 7175456 INFO  (jetty-closer-2934-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@77d2e080: rootName = solr_45447, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@65390ae4
   [junit4]   2> 7175493 INFO  (jetty-closer-2934-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 7175493 INFO  (jetty-closer-2934-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3955106c: rootName = solr_45447, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@25a2fea6
   [junit4]   2> 7175497 INFO  (jetty-closer-2934-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 7175497 INFO  (jetty-closer-2934-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@498738df: rootName = solr_45447, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@1bb1f9ab
   [junit4]   2> 7175514 INFO  (jetty-closer-2934-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 7175579 INFO  (jetty-closer-2934-thread-2) [    ] o.a.s.c.Overseer Overseer (id=98733110441017361-127.0.0.1:38557_solr-n_0000000005) closing
   [junit4]   2> 7175579 INFO  (OverseerStateUpdate-98733110441017361-127.0.0.1:38557_solr-n_0000000005) [n:127.0.0.1:38557_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:38557_solr
   [junit4]   2> 7175602 INFO  (zkCallback-2945-thread-15-processing-n:127.0.0.1:38557_solr) [n:127.0.0.1:38557_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 7175615 INFO  (jetty-closer-2934-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1df453a9{/solr,null,UNAVAILABLE}
   [junit4]   2> 7177079 WARN  (zkCallback-2977-thread-2-processing-n:127.0.0.1:45447_solr) [n:127.0.0.1:45447_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 7177080 INFO  (jetty-closer-2934-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5971864a{/solr,null,UNAVAILABLE}
   [junit4]   2> 7177129 ERROR (SUITE-TestTlogReplica-seed#[C48A3674D8CF3C7D]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 7177149 INFO  (SUITE-TestTlogReplica-seed#[C48A3674D8CF3C7D]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46580 46580
   [junit4]   2> 7177364 INFO  (Thread-5828) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46580 46580
   [junit4]   2> 7177366 WARN  (Thread-5828) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 21 /solr/configs/conf
   [junit4]   2> 9 /solr/aliases.json
   [junit4]   2> 7 /solr/security.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 42 /solr/collections/tlog_replica_test_create_delete/state.json
   [junit4]   2> 28 /solr/collections/tlog_replica_test_kill_leader/state.json
   [junit4]   2> 28 /solr/collections/tlog_replica_test_kill_tlog_replica/state.json
   [junit4]   2> 27 /solr/collections/tlog_replica_test_remove_leader/state.json
   [junit4]   2> 27 /solr/collections/tlog_replica_test_add_remove_tlog_replica/state.json
   [junit4]   2> 26 /solr/collections/tlog_replica_test_basic_leader_election/state.json
   [junit4]   2> 15 /solr/collections/tlog_replica_test_out_of_order_db_qwith_in_place_updates/state.json
   [junit4]   2> 15 /solr/collections/tlog_replica_test_delete_by_id/state.json
   [junit4]   2> 14 /solr/collections/tlog_replica_test_recovery/state.json
   [junit4]   2> 13 /solr/collections/tlog_replica_test_real_time_get/state.json
   [junit4]   2> 13 /solr/collections/tlog_replica_test_only_leader_indexes/state.json
   [junit4]   2> 9 /solr/clusterstate.json
   [junit4]   2> 9 /solr/clusterprops.json
   [junit4]   2> 6 /solr/collections/tlog_replica_test_add_docs/state.json
   [junit4]   2> 3 /solr/overseer_elect/election/98733110441017350-127.0.0.1:38557_solr-n_0000000000
   [junit4]   2> 2 /solr/collections/tlog_replica_test_create_delete/leader_elect/shard1/election/98733110441017350-core_node11-n_0000000001
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 9 /solr/collections
   [junit4]   2> 5 /solr/live_nodes
   [junit4]   2> 3 /solr/overseer/queue
   [junit4]   2> 3 /solr/overseer/collection-queue-work
   [junit4]   2> 3 /solr/overseer/queue-work
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.TestTlogReplica_C48A3674D8CF3C7D-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {foo=BlockTreeOrds(blocksize=128), id=FSTOrd50, foo_s=BlockTreeOrds(blocksize=128)}, docValues:{_version_=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=1105, maxMBSortInHeap=6.119892758616411, sim=RandomSimilarity(queryNorm=true): {}, locale=es-CL, timezone=Africa/Banjul
   [junit4]   2> NOTE: Linux 4.4.0-83-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=59211896,total=515375104
   [junit4]   2> NOTE: All tests run in this JVM: [TestConfigSets, URLClassifyProcessorTest, OverseerTest, DateFieldTest, HdfsLockFactoryTest, HardAutoCommitTest, ShufflingReplicaListTransformerTest, PrimitiveFieldTypeTest, AutoScalingHandlerTest, SolrJmxReporterCloudTest, TestMaxTokenLenTokenizer, HdfsCollectionsAPIDistributedZkTest, SearchHandlerTest, TestNumericTerms32, DataDrivenBlockJoinTest, TestSmileRequest, TestRawTransformer, CircularListTest, TestOrdValues, HdfsThreadLeakTest, TestClusterProperties, TestRequestStatusCollectionAPI, GraphQueryTest, OverriddenZkACLAndCredentialsProvidersTest, HdfsRecoveryZkTest, HLLSerializationTest, TestPayloadScoreQParserPlugin, TestSolrCloudWithKerberosAlt, SolrInfoBeanTest, HdfsDirectoryFactoryTest, TestSolrCoreProperties, ReturnFieldsTest, BadIndexSchemaTest, DeleteShardTest, TestMaxScoreQueryParser, LeaderFailoverAfterPartitionTest, SmileWriterTest, DeleteLastCustomShardedReplicaTest, QueryEqualityTest, CloudMLTQParserTest, TestDynamicFieldCollectionResource, XsltUpdateRequestHandlerTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestDownShardTolerantSearch, AssignTest, TestEmbeddedSolrServerSchemaAPI, TestSolrCLIRunExample, TestDFISimilarityFactory, TestReload, TestCloudPseudoReturnFields, InfoHandlerTest, CoreSorterTest, JavabinLoaderTest, TestLeaderElectionWithEmptyReplica, BlockJoinFacetDistribTest, TestSort, CloudExitableDirectoryReaderTest, AutoCommitTest, TestComplexPhraseLeadingWildcard, TestCryptoKeys, PingRequestHandlerTest, TestHashQParserPlugin, TestTrieFacet, TestConfigReload, ResponseLogComponentTest, TestDFRSimilarityFactory, TestDocBasedVersionConstraints, NumericFieldsTest, TestFieldResource, MinimalSchemaTest, TestJavabinTupleStreamParser, TestCodecSupport, SignatureUpdateProcessorFactoryTest, PreAnalyzedFieldTest, FacetPivotSmallTest, UnloadDistributedZkTest, TestSolrXml, SolrCoreCheckLockOnStartupTest, ReplaceNodeTest, VersionInfoTest, TermVectorComponentDistributedTest, MoveReplicaHDFSTest, ConcurrentDeleteAndCreateCollectionTest, NotRequiredUniqueKeyTest, HdfsWriteToMultipleCollectionsTest, SortByFunctionTest, TestApiFramework, SolrMetricsIntegrationTest, TestExportWriter, TestRestManager, StatsComponentTest, TestInPlaceUpdatesDistrib, SuggesterFSTTest, SimplePostToolTest, SolrCloudReportersTest, TestRecovery, TestSlowCompositeReaderWrapper, TestCoreAdminApis, DistributedQueryComponentCustomSortTest, SuggesterTest, CheckHdfsIndexTest, BadCopyFieldTest, DirectUpdateHandlerOptimizeTest, TestZkChroot, TestInitParams, TestNumericRangeQuery32, MetricUtilsTest, TestSubQueryTransformerDistrib, TestRebalanceLeaders, TestGroupingSearch, RegexBoostProcessorTest, TestSchemalessBufferedUpdates, OverseerRolesTest, FullSolrCloudDistribCmdsTest, TestAuthorizationFramework, AddBlockUpdateTest, BasicDistributedZkTest, TestUpdate, TestWordDelimiterFilterFactory, IndexBasedSpellCheckerTest, LukeRequestHandlerTest, DocumentAnalysisRequestHandlerTest, FileBasedSpellCheckerTest, TermVectorComponentTest, RequiredFieldsTest, IndexSchemaRuntimeFieldTest, TestCSVResponseWriter, TestAnalyzedSuggestions, HighlighterConfigTest, TestQuerySenderListener, CopyFieldTest, TestSolrDeletionPolicy2, MultiTermTest, TestBinaryField, OutputWriterTest, TestPostingsSolrHighlighter, TestLMJelinekMercerSimilarityFactory, TestFastWriter, OpenExchangeRatesOrgProviderTest, DateMathParserTest, RAMDirectoryFactoryTest, TestUtils, DistributedMLTComponentTest, SolrTestCaseJ4Test, TestCrossCoreJoin, TestDistributedMissingSort, TestSimpleTrackingShardHandler, CdcrBootstrapTest, CollectionsAPISolrJTest, DistribJoinFromCollectionTest, HealthCheckHandlerTest, HttpPartitionTest, RemoteQueryErrorTest, SSLMigrationTest, ShardSplitTest, TestTlogReplica]
   [junit4] Completed [559/733 (3!)] on J2 in 363.91s, 13 tests, 4 errors <<< FAILURES!

[...truncated 47250 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-master - Build # 2107 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/2107/

4 tests failed.
FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
The partitioned replica did not get marked down expected:<[down]> but was:<[active]>

Stack Trace:
org.junit.ComparisonFailure: The partitioned replica did not get marked down expected:<[down]> but was:<[active]>
        at __randomizedtesting.SeedInfo.seed([20D2F918F55B6BF5:A886C6C25BA7060D]:0)
        at org.junit.Assert.assertEquals(Assert.java:125)
        at org.apache.solr.cloud.HttpPartitionTest.testMinRf(HttpPartitionTest.java:241)
        at org.apache.solr.cloud.HttpPartitionTest.test(HttpPartitionTest.java:127)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.RollingRestartTest.test

Error Message:
KeeperErrorCode = Session expired for /live_nodes

Stack Trace:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /live_nodes
        at __randomizedtesting.SeedInfo.seed([20D2F918F55B6BF5:A886C6C25BA7060D]:0)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532)
        at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:340)
        at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:337)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:337)
        at org.apache.solr.common.cloud.ZkStateReader.refreshLiveNodes(ZkStateReader.java:674)
        at org.apache.solr.common.cloud.ZkStateReader.updateLiveNodes(ZkStateReader.java:344)
        at org.apache.solr.cloud.RollingRestartTest.restartWithRolesTest(RollingRestartTest.java:89)
        at org.apache.solr.cloud.RollingRestartTest.test(RollingRestartTest.java:52)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestStressInPlaceUpdates.stressTest

Error Message:
Captured an uncaught exception in thread: Thread[id=19438, name=WRITER1, state=RUNNABLE, group=TGRP-TestStressInPlaceUpdates]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=19438, name=WRITER1, state=RUNNABLE, group=TGRP-TestStressInPlaceUpdates]
Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:44259/_mf/md/collection1: Cannot talk to ZooKeeper - Updates are disabled.
        at __randomizedtesting.SeedInfo.seed([20D2F918F55B6BF5]:0)
        at org.apache.solr.cloud.TestStressInPlaceUpdates$1.run(TestStressInPlaceUpdates.java:306)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:44259/_mf/md/collection1: Cannot talk to ZooKeeper - Updates are disabled.
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)
        at org.apache.solr.cloud.TestStressInPlaceUpdates$1.run(TestStressInPlaceUpdates.java:169)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testOnlyLeaderIndexes

Error Message:
expected:<2> but was:<4>

Stack Trace:
java.lang.AssertionError: expected:<2> but was:<4>
        at __randomizedtesting.SeedInfo.seed([20D2F918F55B6BF5:3CD3849580FE1566]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.cloud.TestTlogReplica.assertCopyOverOldUpdates(TestTlogReplica.java:909)
        at org.apache.solr.cloud.TestTlogReplica.testOnlyLeaderIndexes(TestTlogReplica.java:501)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12105 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/init-core-data-001
   [junit4]   2> 3340306 WARN  (SUITE-HttpPartitionTest-seed#[20D2F918F55B6BF5]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 3340306 INFO  (SUITE-HttpPartitionTest-seed#[20D2F918F55B6BF5]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 3340307 INFO  (SUITE-HttpPartitionTest-seed#[20D2F918F55B6BF5]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 3340307 INFO  (SUITE-HttpPartitionTest-seed#[20D2F918F55B6BF5]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 3340307 INFO  (SUITE-HttpPartitionTest-seed#[20D2F918F55B6BF5]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /os/zk
   [junit4]   2> 3340344 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3340344 INFO  (Thread-2149) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3340344 INFO  (Thread-2149) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3340403 ERROR (Thread-2149) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3340447 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.ZkTestServer start zk server on port:50412
   [junit4]   2> 3340484 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3340488 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 3340492 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3340496 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3340499 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3340503 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3340507 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3340547 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3340551 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3340552 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3340553 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3340642 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 3342414 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 3342432 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6720c2b0{/os/zk,null,AVAILABLE}
   [junit4]   2> 3342432 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@58b89a3c{HTTP/1.1,[http/1.1]}{127.0.0.1:43305}
   [junit4]   2> 3342432 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.e.j.s.Server Started @3365484ms
   [junit4]   2> 3342432 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/tempDir-001/control/data, replicaType=NRT, hostContext=/os/zk, hostPort=50333, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/control-001/cores}
   [junit4]   2> 3342433 ERROR (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3342433 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 3342433 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3342433 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 3342433 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-28T12:30:44.283Z
   [junit4]   2> 3342473 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3342473 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/control-001/solr.xml
   [junit4]   2> 3342494 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3342528 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50412/solr
   [junit4]   2> 3343502 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3344242 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:50333_os%2Fzk
   [junit4]   2> 3344273 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.Overseer Overseer (id=98736658330484740-127.0.0.1:50333_os%2Fzk-n_0000000000) starting
   [junit4]   2> 3344392 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50333_os%2Fzk
   [junit4]   2> 3344420 INFO  (zkCallback-2089-thread-1-processing-n:127.0.0.1:50333_os%2Fzk) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3345218 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812
   [junit4]   2> 3345772 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812
   [junit4]   2> 3345772 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812
   [junit4]   2> 3345773 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/control-001/cores
   [junit4]   2> 3346568 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3346568 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:50412/solr ready
   [junit4]   2> 3346569 INFO  (SocketProxy-Acceptor-50333) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=52651,localport=50333], receiveBufferSize:531000
   [junit4]   2> 3346575 INFO  (SocketProxy-Acceptor-50333) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43305,localport=44561], receiveBufferSize=530904
   [junit4]   2> 3346624 INFO  (qtp1537595351-11216) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:50333_os%252Fzk&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3347240 INFO  (OverseerThreadFactory-2593-thread-1-processing-n:127.0.0.1:50333_os%2Fzk) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 3347860 INFO  (SocketProxy-Acceptor-50333) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=52653,localport=50333], receiveBufferSize:531000
   [junit4]   2> 3347863 INFO  (SocketProxy-Acceptor-50333) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43305,localport=44563], receiveBufferSize=530904
   [junit4]   2> 3347888 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3347888 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3347995 INFO  (zkCallback-2089-thread-1-processing-n:127.0.0.1:50333_os%2Fzk) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3348955 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 3349001 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 3351992 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3352084 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 3352085 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812
   [junit4]   2> 3352085 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3352085 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 3352243 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=28, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3352251 WARN  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3352799 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3352799 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3352801 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3352801 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3352807 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=15, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3352808 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3577b7c[control_collection_shard1_replica_n1] main]
   [junit4]   2> 3352809 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3352809 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3352810 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3352811 INFO  (searcherExecutor-2596-thread-1-processing-n:127.0.0.1:50333_os%2Fzk x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@3577b7c[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3352811 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579786546353012736
   [junit4]   2> 3352834 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3352834 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3352834 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:50333/os/zk/control_collection_shard1_replica_n1/
   [junit4]   2> 3352834 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3352834 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:50333/os/zk/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 3352834 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3352836 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:50333/os/zk/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 3352860 INFO  (zkCallback-2089-thread-1-processing-n:127.0.0.1:50333_os%2Fzk) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3352987 INFO  (zkCallback-2089-thread-1-processing-n:127.0.0.1:50333_os%2Fzk) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3353031 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3353032 INFO  (qtp1537595351-11217) [n:127.0.0.1:50333_os%2Fzk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=5143
   [junit4]   2> 3353444 INFO  (qtp1537595351-11216) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 3353447 INFO  (zkCallback-2089-thread-2-processing-n:127.0.0.1:50333_os%2Fzk) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3353693 INFO  (OverseerCollectionConfigSetProcessor-98736658330484740-127.0.0.1:50333_os%2Fzk-n_0000000000) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3354455 INFO  (qtp1537595351-11216) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:50333_os%252Fzk&wt=javabin&version=2} status=0 QTime=7831
   [junit4]   2> 3354596 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3354596 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:50412/solr ready
   [junit4]   2> 3354596 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3354599 INFO  (SocketProxy-Acceptor-50333) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=52668,localport=50333], receiveBufferSize:531000
   [junit4]   2> 3354604 INFO  (SocketProxy-Acceptor-50333) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43305,localport=44578], receiveBufferSize=530904
   [junit4]   2> 3354645 INFO  (qtp1537595351-11219) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3354674 INFO  (OverseerThreadFactory-2593-thread-2-processing-n:127.0.0.1:50333_os%2Fzk) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 3354674 WARN  (OverseerThreadFactory-2593-thread-2-processing-n:127.0.0.1:50333_os%2Fzk) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 3354792 INFO  (qtp1537595351-11219) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 3354792 INFO  (qtp1537595351-11219) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=147
   [junit4]   2> 3355606 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-1-001 of type NRT
   [junit4]   2> 3355606 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 3355664 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1ab42ef9{/os/zk,null,AVAILABLE}
   [junit4]   2> 3355664 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@c044b76{HTTP/1.1,[http/1.1]}{127.0.0.1:39235}
   [junit4]   2> 3355664 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.e.j.s.Server Started @3378716ms
   [junit4]   2> 3355664 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/os/zk, hostPort=60228, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-1-001/cores}
   [junit4]   2> 3355665 ERROR (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3355665 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 3355665 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3355665 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 3355665 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-28T12:30:57.515Z
   [junit4]   2> 3355893 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3355894 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-1-001/solr.xml
   [junit4]   2> 3355913 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3356008 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50412/solr
   [junit4]   2> 3356315 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:60228_os%2Fzk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3356316 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:60228_os%2Fzk    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3356317 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:60228_os%2Fzk    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60228_os%2Fzk
   [junit4]   2> 3356318 INFO  (zkCallback-2096-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3356318 INFO  (zkCallback-2089-thread-2-processing-n:127.0.0.1:50333_os%2Fzk) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3356333 INFO  (zkCallback-2101-thread-1-processing-n:127.0.0.1:60228_os%2Fzk) [n:127.0.0.1:60228_os%2Fzk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3356692 INFO  (OverseerCollectionConfigSetProcessor-98736658330484740-127.0.0.1:50333_os%2Fzk-n_0000000000) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3357227 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:60228_os%2Fzk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812
   [junit4]   2> 3357329 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:60228_os%2Fzk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812
   [junit4]   2> 3357329 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:60228_os%2Fzk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812
   [junit4]   2> 3357331 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:60228_os%2Fzk    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-1-001/cores
   [junit4]   2> 3357468 INFO  (qtp1537595351-11220) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:60228_os%252Fzk&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3358189 INFO  (OverseerThreadFactory-2593-thread-3-processing-n:127.0.0.1:50333_os%2Fzk) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:60228_os%2Fzk for creating new replica
   [junit4]   2> 3358197 INFO  (SocketProxy-Acceptor-60228) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=59994,localport=60228], receiveBufferSize:531000
   [junit4]   2> 3358256 INFO  (SocketProxy-Acceptor-60228) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=39235,localport=33521], receiveBufferSize=530904
   [junit4]   2> 3358611 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3358611 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3358740 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 3358975 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema [collection1_shard2_replica_n41] Schema name=test
   [junit4]   2> 3360686 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3360784 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n41' using configuration from collection collection1, trusted=true
   [junit4]   2> 3360784 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n41' (registry 'solr.core.collection1.shard2.replica_n41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812
   [junit4]   2> 3360784 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3360784 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [[collection1_shard2_replica_n41] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-1-001/cores/collection1_shard2_replica_n41], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-1-001/cores/collection1_shard2_replica_n41/data/]
   [junit4]   2> 3360832 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=28, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3360849 WARN  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3361202 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3361202 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3361212 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3361212 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3361231 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=15, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3361235 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.SolrIndexSearcher Opening [Searcher@297159e1[collection1_shard2_replica_n41] main]
   [junit4]   2> 3361236 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3361236 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3361237 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3361237 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579786555188314112
   [junit4]   2> 3361253 INFO  (searcherExecutor-2607-thread-1-processing-n:127.0.0.1:60228_os%2Fzk x:collection1_shard2_replica_n41 s:shard2 c:collection1) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [collection1_shard2_replica_n41] Registered new searcher Searcher@297159e1[collection1_shard2_replica_n41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3361272 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3361272 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3361272 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:60228/os/zk/collection1_shard2_replica_n41/
   [junit4]   2> 3361272 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3361272 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy http://127.0.0.1:60228/os/zk/collection1_shard2_replica_n41/ has no replicas
   [junit4]   2> 3361272 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3361285 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:60228/os/zk/collection1_shard2_replica_n41/ shard2
   [junit4]   2> 3361401 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3361402 INFO  (qtp680866672-11268) [n:127.0.0.1:60228_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2791
   [junit4]   2> 3361404 INFO  (qtp1537595351-11220) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:60228_os%252Fzk&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=3936
   [junit4]   2> 3361503 INFO  (OverseerCollectionConfigSetProcessor-98736658330484740-127.0.0.1:50333_os%2Fzk-n_0000000000) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3362157 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-2-001 of type NRT
   [junit4]   2> 3362158 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 3362216 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7249e0e2{/os/zk,null,AVAILABLE}
   [junit4]   2> 3362216 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@77039fc9{HTTP/1.1,[http/1.1]}{127.0.0.1:58815}
   [junit4]   2> 3362216 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.e.j.s.Server Started @3385268ms
   [junit4]   2> 3362216 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/os/zk, hostPort=33059, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-2-001/cores}
   [junit4]   2> 3362216 ERROR (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3362543 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 3362543 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3362543 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 3362543 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-28T12:31:04.393Z
   [junit4]   2> 3362653 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3362653 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-2-001/solr.xml
   [junit4]   2> 3362657 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3362712 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50412/solr
   [junit4]   2> 3362755 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:33059_os%2Fzk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3362775 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:33059_os%2Fzk    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3362776 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:33059_os%2Fzk    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33059_os%2Fzk
   [junit4]   2> 3362776 INFO  (zkCallback-2096-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3362791 INFO  (zkCallback-2101-thread-1-processing-n:127.0.0.1:60228_os%2Fzk) [n:127.0.0.1:60228_os%2Fzk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3362792 INFO  (zkCallback-2089-thread-1-processing-n:127.0.0.1:50333_os%2Fzk) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3362895 INFO  (zkCallback-2107-thread-1-processing-n:127.0.0.1:33059_os%2Fzk) [n:127.0.0.1:33059_os%2Fzk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3363722 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:33059_os%2Fzk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812
   [junit4]   2> 3363886 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:33059_os%2Fzk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812
   [junit4]   2> 3363886 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:33059_os%2Fzk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812
   [junit4]   2> 3363923 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:33059_os%2Fzk    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-2-001/cores
   [junit4]   2> 3364308 INFO  (SocketProxy-Acceptor-33059) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=56477,localport=33059], receiveBufferSize:531000
   [junit4]   2> 3364315 INFO  (SocketProxy-Acceptor-33059) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=58815,localport=35901], receiveBufferSize=530904
   [junit4]   2> 3364357 INFO  (qtp164302342-11301) [n:127.0.0.1:33059_os%2Fzk    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:33059_os%252Fzk&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3364408 INFO  (OverseerThreadFactory-2593-thread-4-processing-n:127.0.0.1:50333_os%2Fzk) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:33059_os%2Fzk for creating new replica
   [junit4]   2> 3364423 INFO  (SocketProxy-Acceptor-33059) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=56479,localport=33059], receiveBufferSize:531000
   [junit4]   2> 3364424 INFO  (SocketProxy-Acceptor-33059) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=58815,localport=35903], receiveBufferSize=530904
   [junit4]   2> 3364453 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3364453 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3365530 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 3365656 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.IndexSchema [collection1_shard1_replica_n43] Schema name=test
   [junit4]   2> 3366467 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3366984 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n43' using configuration from collection collection1, trusted=true
   [junit4]   2> 3366984 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n43' (registry 'solr.core.collection1.shard1.replica_n43') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812
   [junit4]   2> 3366984 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3366984 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrCore [[collection1_shard1_replica_n43] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-2-001/cores/collection1_shard1_replica_n43], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-2-001/cores/collection1_shard1_replica_n43/data/]
   [junit4]   2> 3367012 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=28, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3367039 WARN  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3367459 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3367459 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3367472 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3367472 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3367491 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=15, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3367579 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.SolrIndexSearcher Opening [Searcher@231a4690[collection1_shard1_replica_n43] main]
   [junit4]   2> 3367580 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3367580 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3367581 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3367604 INFO  (searcherExecutor-2618-thread-1-processing-n:127.0.0.1:33059_os%2Fzk x:collection1_shard1_replica_n43 s:shard1 c:collection1) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrCore [collection1_shard1_replica_n43] Registered new searcher Searcher@231a4690[collection1_shard1_replica_n43] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3367623 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579786561884520448
   [junit4]   2> 3367675 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3367675 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3367675 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33059/os/zk/collection1_shard1_replica_n43/
   [junit4]   2> 3367675 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3367675 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy http://127.0.0.1:33059/os/zk/collection1_shard1_replica_n43/ has no replicas
   [junit4]   2> 3367675 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3367695 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33059/os/zk/collection1_shard1_replica_n43/ shard1
   [junit4]   2> 3367807 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3367809 INFO  (qtp164302342-11299) [n:127.0.0.1:33059_os%2Fzk c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3355
   [junit4]   2> 3367818 INFO  (qtp164302342-11301) [n:127.0.0.1:33059_os%2Fzk    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:33059_os%252Fzk&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=3461
   [junit4]   2> 3370794 INFO  (OverseerCollectionConfigSetProcessor-98736658330484740-127.0.0.1:50333_os%2Fzk-n_0000000000) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3371876 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-3-001 of type NRT
   [junit4]   2> 3371877 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 3372047 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@bdabbf0{/os/zk,null,AVAILABLE}
   [junit4]   2> 3372048 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2dc7658{HTTP/1.1,[http/1.1]}{127.0.0.1:41715}
   [junit4]   2> 3372048 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.e.j.s.Server Started @3395100ms
   [junit4]   2> 3372048 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/os/zk, hostPort=41654, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-3-001/cores}
   [junit4]   2> 3372048 ERROR (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3372056 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 3372056 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3372056 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 3372056 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-28T12:31:13.906Z
   [junit4]   2> 3372132 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3372132 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-3-001/solr.xml
   [junit4]   2> 3372168 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3372176 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50412/solr
   [junit4]   2> 3372225 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:41654_os%2Fzk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 3372226 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:41654_os%2Fzk    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3372236 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:41654_os%2Fzk    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41654_os%2Fzk
   [junit4]   2> 3372243 INFO  (zkCallback-2101-thread-1-processing-n:127.0.0.1:60228_os%2Fzk) [n:127.0.0.1:60228_os%2Fzk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3372243 INFO  (zkCallback-2107-thread-1-processing-n:127.0.0.1:33059_os%2Fzk) [n:127.0.0.1:33059_os%2Fzk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3372243 INFO  (zkCallback-2089-thread-2-processing-n:127.0.0.1:50333_os%2Fzk) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3372243 INFO  (zkCallback-2096-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3372383 INFO  (zkCallback-2113-thread-1-processing-n:127.0.0.1:41654_os%2Fzk) [n:127.0.0.1:41654_os%2Fzk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3374756 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:41654_os%2Fzk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812
   [junit4]   2> 3374765 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:41654_os%2Fzk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812
   [junit4]   2> 3374765 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:41654_os%2Fzk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812
   [junit4]   2> 3374792 INFO  (TEST-HttpPartitionTest.test-seed#[20D2F918F55B6BF5]) [n:127.0.0.1:41654_os%2Fzk    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-3-001/cores
   [junit4]   2> 3374917 INFO  (qtp164302342-11302) [n:127.0.0.1:33059_os%2Fzk    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:41654_os%252Fzk&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3374928 INFO  (OverseerThreadFactory-2593-thread-5-processing-n:127.0.0.1:50333_os%2Fzk) [n:127.0.0.1:50333_os%2Fzk    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:41654_os%2Fzk for creating new replica
   [junit4]   2> 3374940 INFO  (SocketProxy-Acceptor-41654) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=38481,localport=41654], receiveBufferSize:531000
   [junit4]   2> 3374951 INFO  (SocketProxy-Acceptor-41654) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=41715,localport=47580], receiveBufferSize=530904
   [junit4]   2> 3375008 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n45&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3375008 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3375055 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 3375092 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.s.IndexSchema [collection1_shard2_replica_n45] Schema name=test
   [junit4]   2> 3375886 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3375961 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n45' using configuration from collection collection1, trusted=true
   [junit4]   2> 3375961 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n45' (registry 'solr.core.collection1.shard2.replica_n45') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7197c812
   [junit4]   2> 3375961 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3375961 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.c.SolrCore [[collection1_shard2_replica_n45] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-3-001/cores/collection1_shard2_replica_n45], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest_20D2F918F55B6BF5-001/shard-3-001/cores/collection1_shard2_replica_n45/data/]
   [junit4]   2> 3376008 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=28, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3376037 WARN  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3376335 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3376336 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3376340 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3376340 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3376367 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=15, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3376383 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.s.SolrIndexSearcher Opening [Searcher@15e8a40[collection1_shard2_replica_n45] main]
   [junit4]   2> 3376384 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3376384 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3376385 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3376385 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579786571072143360
   [junit4]   2> 3376388 INFO  (searcherExecutor-2629-thread-1-processing-n:127.0.0.1:41654_os%2Fzk x:collection1_shard2_replica_n45 s:shard2 c:collection1) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.c.SolrCore [collection1_shard2_replica_n45] Registered new searcher Searcher@15e8a40[collection1_shard2_replica_n45] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3376443 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.c.ZkController Core needs to recover:collection1_shard2_replica_n45
   [junit4]   2> 3376444 INFO  (qtp1970627187-11336) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n45&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1435
   [junit4]   2> 3376460 INFO  (updateExecutor-2110-thread-1-processing-n:127.0.0.1:41654_os%2Fzk x:collection1_shard2_replica_n45 s:shard2 c:collection1) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 3376471 INFO  (recoveryExecutor-2111-thread-1-processing-n:127.0.0.1:41654_os%2Fzk x:collection1_shard2_replica_n45 s:shard2 c:collection1 r:core_node46) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 3376472 INFO  (recoveryExecutor-2111-thread-1-processing-n:127.0.0.1:41654_os%2Fzk x:collection1_shard2_replica_n45 s:shard2 c:collection1 r:core_node46) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 3376472 INFO  (recoveryExecutor-2111-thread-1-processing-n:127.0.0.1:41654_os%2Fzk x:collection1_shard2_replica_n45 s:shard2 c:collection1 r:core_node46) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1_shard2_replica_n45]
   [junit4]   2> 3376472 INFO  (recoveryExecutor-2111-thread-1-processing-n:127.0.0.1:41654_os%2Fzk x:collection1_shard2_replica_n45 s:shard2 c:collection1 r:core_node46) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 3376472 INFO  (recoveryExecutor-2111-thread-1-processing-n:127.0.0.1:41654_os%2Fzk x:collection1_shard2_replica_n45 s:shard2 c:collection1 r:core_node46) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.c.RecoveryStrategy Publishing state of core [collection1_shard2_replica_n45] as recovering, leader is [http://127.0.0.1:60228/os/zk/collection1_shard2_replica_n41/] and I am [http://127.0.0.1:41654/os/zk/collection1_shard2_replica_n45/]
   [junit4]   2> 3376474 INFO  (recoveryExecutor-2111-thread-1-processing-n:127.0.0.1:41654_os%2Fzk x:collection1_shard2_replica_n45 s:shard2 c:collection1 r:core_node46) [n:127.0.0.1:41654_os%2Fzk c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:60228/os/zk]; [WaitForState: action=PREPRECOVERY&core=collection1_shard2_replica_n41&nodeName=127.0.0.1:41654_os%252Fzk&coreNodeName=core_node46&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 3376474 INFO  (SocketProxy-Acceptor-60228) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=60016,localport=60228], receiveB

[...truncated too long message...]

processing-n:127.0.0.1:57131_solr) [n:127.0.0.1:57131_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 8546531 INFO  (jetty-closer-4318-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4e332af9{/solr,null,UNAVAILABLE}
   [junit4]   2> 8546559 ERROR (SUITE-TestTlogReplica-seed#[20D2F918F55B6BF5]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 8546560 INFO  (SUITE-TestTlogReplica-seed#[20D2F918F55B6BF5]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54503 54503
   [junit4]   2> 8546651 INFO  (Thread-4923) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54503 54503
   [junit4]   2> 8546651 WARN  (Thread-4923) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 33 /solr/configs/conf
   [junit4]   2> 11 /solr/aliases.json
   [junit4]   2> 9 /solr/security.json
   [junit4]   2> 3 /solr/clusterprops.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 52 /solr/collections/tlog_replica_test_create_delete/state.json
   [junit4]   2> 48 /solr/collections/tlog_replica_test_recovery/state.json
   [junit4]   2> 32 /solr/collections/tlog_replica_test_add_remove_tlog_replica/state.json
   [junit4]   2> 29 /solr/collections/tlog_replica_test_out_of_order_db_qwith_in_place_updates/state.json
   [junit4]   2> 27 /solr/collections/tlog_replica_test_remove_leader/state.json
   [junit4]   2> 26 /solr/collections/tlog_replica_test_kill_leader/state.json
   [junit4]   2> 26 /solr/collections/tlog_replica_test_basic_leader_election/state.json
   [junit4]   2> 26 /solr/collections/tlog_replica_test_kill_tlog_replica/state.json
   [junit4]   2> 16 /solr/collections/tlog_replica_test_only_leader_indexes/state.json
   [junit4]   2> 15 /solr/collections/tlog_replica_test_real_time_get/state.json
   [junit4]   2> 15 /solr/collections/tlog_replica_test_delete_by_id/state.json
   [junit4]   2> 11 /solr/clusterprops.json
   [junit4]   2> 11 /solr/clusterstate.json
   [junit4]   2> 6 /solr/collections/tlog_replica_test_add_docs/state.json
   [junit4]   2> 5 /solr/overseer_elect/election/98736982226698252-127.0.0.1:51326_solr-n_0000000002
   [junit4]   2> 4 /solr/collections/tlog_replica_test_recovery/leader_elect/shard1/election/98736982226698252-core_node4-n_0000000000
   [junit4]   2> 2 /solr/overseer_elect/election/98736982226698246-127.0.0.1:57131_solr-n_0000000000
   [junit4]   2> 2 /solr/collections/tlog_replica_test_create_delete/leader_elect/shard2/election/98736982226698246-core_node13-n_0000000000
   [junit4]   2> 2 /solr/collections/tlog_replica_test_create_delete/leader_elect/shard2/election/98736982226698244-core_node11-n_0000000000
   [junit4]   2> 2 /solr/collections/tlog_replica_test_kill_tlog_replica/leader_elect/shard1/election/98736982226698267-core_node4-n_0000000000
   [junit4]   2> 2 /solr/collections/tlog_replica_test_create_delete/leader_elect/shard1/election/98736982226698244-core_node3-n_0000000000
   [junit4]   2> 2 /solr/collections/tlog_replica_test_create_delete/leader_elect/shard1/election/98736982226698244-core_node7-n_0000000000
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 11 /solr/collections
   [junit4]   2> 4 /solr/live_nodes
   [junit4]   2> 3 /solr/overseer/queue
   [junit4]   2> 3 /solr/overseer/collection-queue-work
   [junit4]   2> 3 /solr/overseer/queue-work
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_20D2F918F55B6BF5-001
   [junit4]   2> Sep 28, 2017 1:57:30 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {foo=BlockTreeOrds(blocksize=128), title_s=BlockTreeOrds(blocksize=128), id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), foo_s=BlockTreeOrds(blocksize=128)}, docValues:{_version_=DocValuesFormat(name=Asserting), id=DocValuesFormat(name=Memory), inplace_updatable_int=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1399, maxMBSortInHeap=6.0348383979168565, sim=RandomSimilarity(queryNorm=true): {}, locale=lv-LV, timezone=America/St_Johns
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=127368944,total=522190848
   [junit4]   2> NOTE: All tests run in this JVM: [SaslZkACLProviderTest, TestReplicationHandlerBackup, TestLegacyFieldReuse, AlternateDirectoryTest, ResourceLoaderTest, BaseCdcrDistributedZkTest, TestRequestForwarding, MoveReplicaTest, LeaderInitiatedRecoveryOnCommitTest, TestSolrQueryResponse, ConfigureRecoveryStrategyTest, TestBackupRepositoryFactory, TestMultiWordSynonyms, SOLR749Test, TestXIncludeConfig, DistributedIntervalFacetingTest, TestPullReplica, CollectionsAPIDistributedZkTest, ForceLeaderTest, BlockCacheTest, DistributedExpandComponentTest, JSONWriterTest, StatsReloadRaceTest, DistributedMLTComponentTest, DistributedFacetPivotWhiteBoxTest, DistanceFunctionTest, TestHdfsCloudBackupRestore, TestSweetSpotSimilarityFactory, TestSQLHandler, TestCollectionsAPIViaSolrCloudCluster, TestMiniSolrCloudClusterSSL, RestartWhileUpdatingTest, UpdateRequestProcessorFactoryTest, AssignBackwardCompatibilityTest, CdcrUpdateLogTest, ReplicaListTransformerTest, ConnectionManagerTest, OverseerTaskQueueTest, TestCustomSort, CollectionsAPISolrJTest, MissingSegmentRecoveryTest, TestSolrCloudWithDelegationTokens, BasicFunctionalityTest, TermVectorComponentTest, DirectSolrSpellCheckerTest, SampleTest, JsonLoaderTest, MetricsConfigTest, TestCustomStream, CacheHeaderTest, TestBM25SimilarityFactory, StatelessScriptUpdateProcessorFactoryTest, TestCopyFieldCollectionResource, TestHdfsUpdateLog, TestFieldTypeResource, HttpSolrCallGetCoreTest, CleanupOldIndexTest, TestUninvertingReader, TestConfigsApi, DocValuesTest, QueryParsingTest, TestSimpleQParserPlugin, SharedFSAutoReplicaFailoverUtilsTest, ConnectionReuseTest, ZkCLITest, TestFileDictionaryLookup, LoggingHandlerTest, SolrCloudExampleTest, TestNumericTokenStream, TestAtomicUpdateErrorCases, MoreLikeThisHandlerTest, HdfsBasicDistributedZk2Test, ShardRoutingTest, ImplicitSnitchTest, TestUniqueKeyFieldResource, TestReloadAndDeleteDocs, DocValuesMultiTest, TestCloudPivotFacet, TestSchemaManager, SolrSlf4jReporterTest, TestCoreContainer, V2StandaloneTest, LeaderElectionContextKeyTest, TestPerFieldSimilarity, TestUnifiedSolrHighlighter, TestCoreDiscovery, HdfsUnloadDistributedZkTest, RankQueryTest, TestLeaderInitiatedRecoveryThread, TestSolrConfigHandler, DistribCursorPagingTest, BadComponentTest, TestRandomRequestDistribution, TestStreamBody, TestFieldCache, FileBasedSpellCheckerTest, TestLRUCache, TestLRUStatsCache, TestStressCloudBlindAtomicUpdates, CoreMergeIndexesAdminHandlerTest, TestBlobHandler, TestManagedSchemaAPI, PKIAuthenticationIntegrationTest, BigEndianAscendingWordSerializerTest, SliceStateTest, ShardRoutingCustomTest, TestCollapseQParserPlugin, TestClusterStateMutator, ClassificationUpdateProcessorIntegrationTest, SoftAutoCommitTest, AtomicUpdateProcessorFactoryTest, ClusterStateUpdateTest, TestInfoStreamLogging, SubstringBytesRefFilterTest, TestRTimerTree, TestMultiValuedNumericRangeQuery, TestUseDocValuesAsStored2, TestJsonRequest, TestSchemaSimilarityResource, TestNestedDocsSort, SystemInfoHandlerTest, DeleteInactiveReplicaTest, ChaosMonkeySafeLeaderTest, SyncSliceTest, OpenCloseCoreStressTest, LeaderElectionIntegrationTest, BasicZkTest, TestReplicationHandler, TestRandomFaceting, ZkSolrClientTest, TestDistributedGrouping, TermVectorComponentDistributedTest, DistributedTermsComponentTest, SolrCoreTest, SpellCheckComponentTest, SolrCmdDistributorTest, QueryElevationComponentTest, TestFiltering, ShowFileRequestHandlerTest, TestExtendedDismaxParser, TestFoldingMultitermQuery, SuggesterTSTTest, TestCSVLoader, NoCacheHeaderTest, SchemaVersionSpecificBehaviorTest, TestWordDelimiterFilterFactory, TestRemoteStreaming, DisMaxRequestHandlerTest, TestWriterPerf, RequiredFieldsTest, FastVectorHighlighterTest, TestJmxIntegration, TestAnalyzedSuggestions, CSVRequestHandlerTest, TestSolrIndexConfig, TestQuerySenderNoQuery, CopyFieldTest, ResponseLogComponentTest, TestSolrDeletionPolicy2, MultiTermTest, TestDocSet, TestSearchPerf, OutputWriterTest, TestSolrCoreProperties, DirectSolrConnectionTest, TestLuceneMatchVersion, TimeZoneUtilsTest, OpenExchangeRatesOrgProviderTest, TestSystemIdResolver, PrimUtilsTest, DOMUtilTest, RAMDirectoryFactoryTest, TestSolrJ, TestUtils, TestDocumentBuilder, UUIDFieldTest, FileUtilsTest, TestRTGBase, TestCursorMarkWithoutUniqueKey, TestHighlightDedupGrouping, TestTolerantSearch, TestEmbeddedSolrServerAdminHandler, TestEmbeddedSolrServerConstructors, TestEmbeddedSolrServerSchemaAPI, ActionThrottleTest, AliasIntegrationTest, AsyncCallRequestStatusResponseTest, CdcrBootstrapTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, ConcurrentDeleteAndCreateCollectionTest, CreateCollectionCleanupTest, DeleteReplicaTest, DocValuesNotIndexedTest, MoveReplicaHDFSTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerRolesTest, SolrCLIZkUtilsTest, SolrXmlInZkTest, TestCloudInspectUtil, TestCloudJSONFacetJoinDomain, TestSizeLimitedDistributedMap, TestSolrCloudWithSecureImpersonation, TestTlogReplica]
   [junit4] Completed [560/733 (4!)] on J1 in 266.91s, 13 tests, 1 failure <<< FAILURES!

[...truncated 47252 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-master - Build # 2108 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/2108/

7 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test

Error Message:
KeeperErrorCode = Session expired for /live_nodes

Stack Trace:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /live_nodes
        at __randomizedtesting.SeedInfo.seed([86FA328456201960:EAE0D5EF8DC7498]:0)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532)
        at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:340)
        at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:337)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:337)
        at org.apache.solr.common.cloud.ZkStateReader.refreshLiveNodes(ZkStateReader.java:674)
        at org.apache.solr.common.cloud.ZkStateReader.updateLiveNodes(ZkStateReader.java:344)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.logReplicaTypesReplicationInfo(AbstractFullDistribZkTestBase.java:2067)
        at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test(ChaosMonkeyNothingIsSafeWithPullReplicasTest.java:312)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest

Error Message:
3 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest:     1) Thread[id=4325, name=zkCallback-670-thread-5-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    2) Thread[id=4211, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=4324, name=zkCallback-670-thread-5-SendThread(127.0.0.1:58006), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 3 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest:
   1) Thread[id=4325, name=zkCallback-670-thread-5-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   2) Thread[id=4211, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=4324, name=zkCallback-670-thread-5-SendThread(127.0.0.1:58006), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
        at __randomizedtesting.SeedInfo.seed([86FA328456201960]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=4324, name=zkCallback-670-thread-5-SendThread(127.0.0.1:58006), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=4324, name=zkCallback-670-thread-5-SendThread(127.0.0.1:58006), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
        at __randomizedtesting.SeedInfo.seed([86FA328456201960]:0)


FAILED:  org.apache.solr.client.solrj.io.graph.GraphExpressionTest.testScoreNodesStream

Error Message:
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: org.apache.solr.common.SolrException: Could not load collection from ZK: collection1

Stack Trace:
java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at __randomizedtesting.SeedInfo.seed([BD335F66E322B0CC:157B8B9AC6577A7F]:0)
        at org.apache.solr.client.solrj.io.graph.GatherNodesStream.read(GatherNodesStream.java:602)
        at org.apache.solr.client.solrj.io.stream.ScoreNodesStream.open(ScoreNodesStream.java:179)
        at org.apache.solr.client.solrj.io.stream.SortStream.open(SortStream.java:169)
        at org.apache.solr.client.solrj.io.graph.GraphExpressionTest.getTuples(GraphExpressionTest.java:930)
        at org.apache.solr.client.solrj.io.graph.GraphExpressionTest.testScoreNodesStream(GraphExpressionTest.java:494)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at org.apache.solr.client.solrj.io.graph.GatherNodesStream.read(GatherNodesStream.java:602)
        at org.apache.solr.client.solrj.io.graph.GatherNodesStream.read(GatherNodesStream.java:520)
        ... 43 more
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.solr.client.solrj.io.graph.GatherNodesStream.read(GatherNodesStream.java:574)
        ... 44 more
Caused by: java.lang.RuntimeException: java.io.IOException: org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at org.apache.solr.client.solrj.io.graph.GatherNodesStream$JoinRunner.call(GatherNodesStream.java:489)
        at org.apache.solr.client.solrj.io.graph.GatherNodesStream$JoinRunner.call(GatherNodesStream.java:408)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more
Caused by: java.io.IOException: org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at org.apache.solr.client.solrj.io.stream.CloudSolrStream.constructStreams(CloudSolrStream.java:383)
        at org.apache.solr.client.solrj.io.stream.CloudSolrStream.open(CloudSolrStream.java:274)
        at org.apache.solr.client.solrj.io.stream.PushBackStream.open(PushBackStream.java:70)
        at org.apache.solr.client.solrj.io.stream.ReducerStream.open(ReducerStream.java:199)
        at org.apache.solr.client.solrj.io.stream.UniqueStream.open(UniqueStream.java:150)
        at org.apache.solr.client.solrj.io.graph.GatherNodesStream$JoinRunner.call(GatherNodesStream.java:478)
        ... 6 more
Caused by: org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1115)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.common.cloud.ClusterState.getCollectionsMap(ClusterState.java:144)
        at org.apache.solr.client.solrj.io.stream.TupleStream.getSlices(TupleStream.java:167)
        at org.apache.solr.client.solrj.io.stream.TupleStream.getShards(TupleStream.java:141)
        at org.apache.solr.client.solrj.io.stream.CloudSolrStream.constructStreams(CloudSolrStream.java:368)
        ... 11 more
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1127)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1113)
        ... 16 more


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.graph.GraphExpressionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.client.solrj.io.graph.GraphExpressionTest:     1) Thread[id=862, name=zkCallback-169-thread-1, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=879, name=zkCallback-169-thread-2, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=866, name=Connection evictor, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=885, name=zkCallback-169-thread-2-EventThread, state=WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    5) Thread[id=884, name=zkCallback-169-thread-2-SendThread(127.0.0.1:51628), state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    6) Thread[id=880, name=zkCallback-169-thread-3, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=859, name=Connection evictor, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    8) Thread[id=865, name=Connection evictor, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    9) Thread[id=991, name=zkCallback-169-thread-5, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   10) Thread[id=883, name=zkCallback-169-thread-4, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.client.solrj.io.graph.GraphExpressionTest:
   1) Thread[id=862, name=zkCallback-169-thread-1, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=879, name=zkCallback-169-thread-2, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=866, name=Connection evictor, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=885, name=zkCallback-169-thread-2-EventThread, state=WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   5) Thread[id=884, name=zkCallback-169-thread-2-SendThread(127.0.0.1:51628), state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   6) Thread[id=880, name=zkCallback-169-thread-3, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=859, name=Connection evictor, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=865, name=Connection evictor, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   9) Thread[id=991, name=zkCallback-169-thread-5, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  10) Thread[id=883, name=zkCallback-169-thread-4, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([BD335F66E322B0CC]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.graph.GraphExpressionTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=862, name=zkCallback-169-thread-1, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=879, name=zkCallback-169-thread-2, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=884, name=zkCallback-169-thread-2-SendThread(127.0.0.1:51628), state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    4) Thread[id=880, name=zkCallback-169-thread-3, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=991, name=zkCallback-169-thread-5, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=883, name=zkCallback-169-thread-4, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=862, name=zkCallback-169-thread-1, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=879, name=zkCallback-169-thread-2, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=884, name=zkCallback-169-thread-2-SendThread(127.0.0.1:51628), state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   4) Thread[id=880, name=zkCallback-169-thread-3, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=991, name=zkCallback-169-thread-5, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=883, name=zkCallback-169-thread-4, state=TIMED_WAITING, group=TGRP-GraphExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([BD335F66E322B0CC]:0)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testOnlyLeaderIndexes

Error Message:
expected:<2> but was:<6>

Stack Trace:
java.lang.AssertionError: expected:<2> but was:<6>
        at __randomizedtesting.SeedInfo.seed([86FA328456201960:9AFB4F09238567F3]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.cloud.TestTlogReplica.assertCopyOverOldUpdates(TestTlogReplica.java:909)
        at org.apache.solr.cloud.TestTlogReplica.testOnlyLeaderIndexes(TestTlogReplica.java:501)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11615 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestTlogReplica
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_86FA328456201960-001/init-core-data-001
   [junit4]   2> 186238 WARN  (SUITE-TestTlogReplica-seed#[86FA328456201960]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=6 numCloses=6
   [junit4]   2> 186239 INFO  (SUITE-TestTlogReplica-seed#[86FA328456201960]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 186240 INFO  (SUITE-TestTlogReplica-seed#[86FA328456201960]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 186240 INFO  (SUITE-TestTlogReplica-seed#[86FA328456201960]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 186241 INFO  (SUITE-TestTlogReplica-seed#[86FA328456201960]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_86FA328456201960-001/tempDir-001
   [junit4]   2> 186241 INFO  (SUITE-TestTlogReplica-seed#[86FA328456201960]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 186256 INFO  (Thread-50) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 186256 INFO  (Thread-50) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 186327 ERROR (Thread-50) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 186395 INFO  (SUITE-TestTlogReplica-seed#[86FA328456201960]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:33491
   [junit4]   2> 186984 INFO  (jetty-launcher-49-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 187019 INFO  (jetty-launcher-49-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 187545 INFO  (jetty-launcher-49-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3f54c1cf{/solr,null,AVAILABLE}
   [junit4]   2> 187558 INFO  (jetty-launcher-49-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4c20079d{HTTP/1.1,[http/1.1]}{127.0.0.1:41256}
   [junit4]   2> 187558 INFO  (jetty-launcher-49-thread-2) [    ] o.e.j.s.Server Started @207719ms
   [junit4]   2> 187566 INFO  (jetty-launcher-49-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41256}
   [junit4]   2> 187566 ERROR (jetty-launcher-49-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 187566 INFO  (jetty-launcher-49-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 187566 INFO  (jetty-launcher-49-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 187566 INFO  (jetty-launcher-49-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 187566 INFO  (jetty-launcher-49-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-29T10:27:58.571Z
   [junit4]   2> 188888 INFO  (jetty-launcher-49-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5888f02c{/solr,null,AVAILABLE}
   [junit4]   2> 188889 INFO  (jetty-launcher-49-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@edcd160{HTTP/1.1,[http/1.1]}{127.0.0.1:60139}
   [junit4]   2> 188889 INFO  (jetty-launcher-49-thread-1) [    ] o.e.j.s.Server Started @209050ms
   [junit4]   2> 188889 INFO  (jetty-launcher-49-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=60139}
   [junit4]   2> 188889 ERROR (jetty-launcher-49-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 188889 INFO  (jetty-launcher-49-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 188889 INFO  (jetty-launcher-49-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 188889 INFO  (jetty-launcher-49-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 188889 INFO  (jetty-launcher-49-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-29T10:27:59.894Z
   [junit4]   2> 189118 INFO  (jetty-launcher-49-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 189238 INFO  (jetty-launcher-49-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 189335 INFO  (jetty-launcher-49-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33491/solr
   [junit4]   2> 189429 INFO  (jetty-launcher-49-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33491/solr
   [junit4]   2> 190598 INFO  (jetty-launcher-49-thread-1) [n:127.0.0.1:60139_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 190608 INFO  (jetty-launcher-49-thread-1) [n:127.0.0.1:60139_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60139_solr
   [junit4]   2> 190624 INFO  (jetty-launcher-49-thread-1) [n:127.0.0.1:60139_solr    ] o.a.s.c.Overseer Overseer (id=98741837972307973-127.0.0.1:60139_solr-n_0000000000) starting
   [junit4]   2> 191353 INFO  (jetty-launcher-49-thread-2) [n:127.0.0.1:41256_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 191358 INFO  (jetty-launcher-49-thread-2) [n:127.0.0.1:41256_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41256_solr
   [junit4]   2> 193103 INFO  (zkCallback-61-thread-1-processing-n:127.0.0.1:41256_solr) [n:127.0.0.1:41256_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 193135 INFO  (zkCallback-60-thread-1-processing-n:127.0.0.1:60139_solr) [n:127.0.0.1:60139_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 193208 INFO  (jetty-launcher-49-thread-1) [n:127.0.0.1:60139_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60139_solr
&n