[JENKINS] Lucene-Solr-7.x-Windows (32bit/jdk1.8.0_172) - Build # 827 - Failure!

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-7.x-Windows (32bit/jdk1.8.0_172) - Build # 827 - Failure!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Windows/827/
Java: 32bit/jdk1.8.0_172 -server -XX:+UseParallelGC

2 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:53507_solr, 127.0.0.1:53508_solr] Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/19)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node3":{           "core":"testSimple1_shard1_replica_n1",           "base_url":"http://127.0.0.1:53507/solr",           "node_name":"127.0.0.1:53507_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node12":{           "core":"testSimple1_shard1_replica_n11",           "base_url":"http://127.0.0.1:53507/solr",           "node_name":"127.0.0.1:53507_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node8":{           "core":"testSimple1_shard2_replica_n6",           "base_url":"http://127.0.0.1:53507/solr",           "node_name":"127.0.0.1:53507_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node10":{           "core":"testSimple1_shard2_replica_n9",           "base_url":"http://127.0.0.1:53513/solr",           "node_name":"127.0.0.1:53513_solr",           "state":"down",           "type":"NRT",           "force_set_state":"false"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:53507_solr, 127.0.0.1:53508_solr]
Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/19)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "core":"testSimple1_shard1_replica_n1",
          "base_url":"http://127.0.0.1:53507/solr",
          "node_name":"127.0.0.1:53507_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node12":{
          "core":"testSimple1_shard1_replica_n11",
          "base_url":"http://127.0.0.1:53507/solr",
          "node_name":"127.0.0.1:53507_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node8":{
          "core":"testSimple1_shard2_replica_n6",
          "base_url":"http://127.0.0.1:53507/solr",
          "node_name":"127.0.0.1:53507_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node10":{
          "core":"testSimple1_shard2_replica_n9",
          "base_url":"http://127.0.0.1:53513/solr",
          "node_name":"127.0.0.1:53513_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([4AE7587C48CC21FA:72547C826F3FF52B]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.preferLocalShardsTest

Error Message:
Response was not received from shards on a single node

Stack Trace:
java.lang.AssertionError: Response was not received from shards on a single node
        at __randomizedtesting.SeedInfo.seed([4A6696CFB3A8092:F86BF1D505ED1768]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.queryWithShardsPreferenceRules(CloudSolrClientTest.java:471)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.preferLocalShardsTest(CloudSolrClientTest.java:420)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14015 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest
   [junit4]   2> Creating dataDir: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\init-core-data-001
   [junit4]   2> 3758030 WARN  (SUITE-AutoAddReplicasIntegrationTest-seed#[4AE7587C48CC21FA]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=27 numCloses=27
   [junit4]   2> 3758030 INFO  (SUITE-AutoAddReplicasIntegrationTest-seed#[4AE7587C48CC21FA]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 3758030 INFO  (SUITE-AutoAddReplicasIntegrationTest-seed#[4AE7587C48CC21FA]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 3758030 INFO  (SUITE-AutoAddReplicasIntegrationTest-seed#[4AE7587C48CC21FA]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 3758036 INFO  (SUITE-AutoAddReplicasIntegrationTest-seed#[4AE7587C48CC21FA]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001
   [junit4]   2> 3758036 INFO  (SUITE-AutoAddReplicasIntegrationTest-seed#[4AE7587C48CC21FA]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3758037 INFO  (Thread-3639) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3758037 INFO  (Thread-3639) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3758040 ERROR (Thread-3639) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3758137 INFO  (SUITE-AutoAddReplicasIntegrationTest-seed#[4AE7587C48CC21FA]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:53503
   [junit4]   2> 3758140 INFO  (zkConnectionManagerCallback-10110-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3758145 INFO  (jetty-launcher-10107-thread-2) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 3758145 INFO  (jetty-launcher-10107-thread-3) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 3758145 INFO  (jetty-launcher-10107-thread-1) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 3758146 INFO  (jetty-launcher-10107-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3758146 INFO  (jetty-launcher-10107-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3758146 INFO  (jetty-launcher-10107-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3758146 INFO  (jetty-launcher-10107-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3758146 INFO  (jetty-launcher-10107-thread-3) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 3758146 INFO  (jetty-launcher-10107-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 3758147 INFO  (jetty-launcher-10107-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3758147 INFO  (jetty-launcher-10107-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3758147 INFO  (jetty-launcher-10107-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7a702b{/solr,null,AVAILABLE}
   [junit4]   2> 3758147 INFO  (jetty-launcher-10107-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 3758147 INFO  (jetty-launcher-10107-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@66e9ac{/solr,null,AVAILABLE}
   [junit4]   2> 3758147 INFO  (jetty-launcher-10107-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1c6957b{/solr,null,AVAILABLE}
   [junit4]   2> 3758148 INFO  (jetty-launcher-10107-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@19373a3{HTTP/1.1,[http/1.1]}{127.0.0.1:53508}
   [junit4]   2> 3758148 INFO  (jetty-launcher-10107-thread-1) [    ] o.e.j.s.Server Started @3758193ms
   [junit4]   2> 3758148 INFO  (jetty-launcher-10107-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53508}
   [junit4]   2> 3758148 INFO  (jetty-launcher-10107-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@52d2fd{HTTP/1.1,[http/1.1]}{127.0.0.1:53513}
   [junit4]   2> 3758148 ERROR (jetty-launcher-10107-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3758148 INFO  (jetty-launcher-10107-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3758148 INFO  (jetty-launcher-10107-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 3758148 INFO  (jetty-launcher-10107-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@e3fb6a{HTTP/1.1,[http/1.1]}{127.0.0.1:53507}
   [junit4]   2> 3758148 INFO  (jetty-launcher-10107-thread-2) [    ] o.e.j.s.Server Started @3758194ms
   [junit4]   2> 3758148 INFO  (jetty-launcher-10107-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53507}
   [junit4]   2> 3758148 INFO  (jetty-launcher-10107-thread-3) [    ] o.e.j.s.Server Started @3758194ms
   [junit4]   2> 3758148 INFO  (jetty-launcher-10107-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3758151 INFO  (jetty-launcher-10107-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3758151 INFO  (jetty-launcher-10107-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53513}
   [junit4]   2> 3758151 INFO  (jetty-launcher-10107-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-09T09:10:07.484Z
   [junit4]   2> 3758151 ERROR (jetty-launcher-10107-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3758152 INFO  (jetty-launcher-10107-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3758152 INFO  (jetty-launcher-10107-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 3758152 INFO  (jetty-launcher-10107-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3758152 INFO  (jetty-launcher-10107-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3758152 INFO  (jetty-launcher-10107-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-09T09:10:07.485Z
   [junit4]   2> 3758153 ERROR (jetty-launcher-10107-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3758153 INFO  (jetty-launcher-10107-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3758153 INFO  (jetty-launcher-10107-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 3758153 INFO  (jetty-launcher-10107-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3758153 INFO  (jetty-launcher-10107-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3758153 INFO  (jetty-launcher-10107-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-09T09:10:07.486Z
   [junit4]   2> 3758155 INFO  (zkConnectionManagerCallback-10112-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3758156 INFO  (jetty-launcher-10107-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3758156 INFO  (zkConnectionManagerCallback-10114-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3758156 INFO  (zkConnectionManagerCallback-10116-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3758156 INFO  (jetty-launcher-10107-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3758156 INFO  (jetty-launcher-10107-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3758169 INFO  (jetty-launcher-10107-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 3758169 INFO  (jetty-launcher-10107-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 3758170 INFO  (jetty-launcher-10107-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 3758170 INFO  (jetty-launcher-10107-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 3758170 INFO  (jetty-launcher-10107-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 3758170 INFO  (jetty-launcher-10107-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 3758170 INFO  (jetty-launcher-10107-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3758171 INFO  (jetty-launcher-10107-thread-3) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3758171 INFO  (jetty-launcher-10107-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3758171 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1007a0037830001, likely client has closed socket
   [junit4]   2> 3758171 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1007a0037830002, likely client has closed socket
   [junit4]   2> 3758172 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1007a0037830003, likely client has closed socket
   [junit4]   2> 3759187 INFO  (jetty-launcher-10107-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53503/solr
   [junit4]   2> 3759190 INFO  (zkConnectionManagerCallback-10120-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3759195 INFO  (zkConnectionManagerCallback-10122-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3759242 INFO  (jetty-launcher-10107-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53503/solr
   [junit4]   2> 3759269 INFO  (zkConnectionManagerCallback-10128-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3759296 INFO  (zkConnectionManagerCallback-10130-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3759318 INFO  (jetty-launcher-10107-thread-2) [n:127.0.0.1:53507_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:53507_solr
   [junit4]   2> 3759320 INFO  (jetty-launcher-10107-thread-2) [n:127.0.0.1:53507_solr    ] o.a.s.c.Overseer Overseer (id=72191735387848711-127.0.0.1:53507_solr-n_0000000000) starting
   [junit4]   2> 3759321 INFO  (jetty-launcher-10107-thread-1) [n:127.0.0.1:53508_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53508_solr
   [junit4]   2> 3759376 INFO  (zkCallback-10129-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3759378 INFO  (zkCallback-10121-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3759378 INFO  (zkConnectionManagerCallback-10137-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3759380 INFO  (jetty-launcher-10107-thread-2) [n:127.0.0.1:53507_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3759381 INFO  (jetty-launcher-10107-thread-2) [n:127.0.0.1:53507_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:53503/solr ready
   [junit4]   2> 3759383 INFO  (jetty-launcher-10107-thread-2) [n:127.0.0.1:53507_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3759383 INFO  (jetty-launcher-10107-thread-2) [n:127.0.0.1:53507_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53507_solr
   [junit4]   2> 3759384 INFO  (jetty-launcher-10107-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53503/solr
   [junit4]   2> 3759385 INFO  (zkCallback-10129-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3759385 INFO  (zkCallback-10121-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3759385 INFO  (zkConnectionManagerCallback-10142-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3759386 INFO  (jetty-launcher-10107-thread-1) [n:127.0.0.1:53508_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3759386 INFO  (jetty-launcher-10107-thread-1) [n:127.0.0.1:53508_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:53503/solr ready
   [junit4]   2> 3759386 INFO  (jetty-launcher-10107-thread-1) [n:127.0.0.1:53508_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3759402 INFO  (jetty-launcher-10107-thread-2) [n:127.0.0.1:53507_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3759417 INFO  (jetty-launcher-10107-thread-1) [n:127.0.0.1:53508_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3
   [junit4]   2> 3759418 INFO  (jetty-launcher-10107-thread-2) [n:127.0.0.1:53507_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3
   [junit4]   2> 3759423 INFO  (jetty-launcher-10107-thread-1) [n:127.0.0.1:53508_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3
   [junit4]   2> 3759424 INFO  (jetty-launcher-10107-thread-1) [n:127.0.0.1:53508_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3
   [junit4]   2> 3759424 INFO  (jetty-launcher-10107-thread-2) [n:127.0.0.1:53507_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3
   [junit4]   2> 3759424 INFO  (jetty-launcher-10107-thread-2) [n:127.0.0.1:53507_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3
   [junit4]   2> 3759426 INFO  (OverseerStateUpdate-72191735387848711-127.0.0.1:53507_solr-n_0000000000) [n:127.0.0.1:53507_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:53507_solr
   [junit4]   2> 3759426 INFO  (jetty-launcher-10107-thread-1) [n:127.0.0.1:53508_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node1\.
   [junit4]   2> 3759426 INFO  (jetty-launcher-10107-thread-2) [n:127.0.0.1:53507_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node2\.
   [junit4]   2> 3759432 DEBUG (OverseerAutoScalingTriggerThread-72191735387848711-127.0.0.1:53507_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 3759433 DEBUG (OverseerAutoScalingTriggerThread-72191735387848711-127.0.0.1:53507_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 3759433 DEBUG (OverseerAutoScalingTriggerThread-72191735387848711-127.0.0.1:53507_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 3759433 DEBUG (OverseerAutoScalingTriggerThread-72191735387848711-127.0.0.1:53507_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 3759437 INFO  (zkCallback-10136-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3759440 DEBUG (OverseerAutoScalingTriggerThread-72191735387848711-127.0.0.1:53507_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:53507_solr, 127.0.0.1:53508_solr]
   [junit4]   2> 3759441 DEBUG (OverseerAutoScalingTriggerThread-72191735387848711-127.0.0.1:53507_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3759441 DEBUG (OverseerAutoScalingTriggerThread-72191735387848711-127.0.0.1:53507_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 3759442 DEBUG (ScheduledTrigger-8357-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3759445 INFO  (zkConnectionManagerCallback-10146-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3759446 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1007a003783000a, likely client has closed socket
   [junit4]   2> 3759448 INFO  (zkConnectionManagerCallback-10150-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3759456 INFO  (jetty-launcher-10107-thread-3) [n:127.0.0.1:53513_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3759460 INFO  (jetty-launcher-10107-thread-3) [n:127.0.0.1:53513_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3759460 INFO  (jetty-launcher-10107-thread-3) [n:127.0.0.1:53513_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53513_solr
   [junit4]   2> 3759494 INFO  (zkCallback-10129-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3759494 INFO  (zkCallback-10121-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3759494 INFO  (zkCallback-10136-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3759505 INFO  (zkCallback-10149-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3759509 INFO  (zkCallback-10141-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3759531 INFO  (zkConnectionManagerCallback-10157-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3759532 INFO  (jetty-launcher-10107-thread-3) [n:127.0.0.1:53513_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 3759533 INFO  (jetty-launcher-10107-thread-3) [n:127.0.0.1:53513_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:53503/solr ready
   [junit4]   2> 3759533 INFO  (jetty-launcher-10107-thread-3) [n:127.0.0.1:53513_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3759568 INFO  (jetty-launcher-10107-thread-3) [n:127.0.0.1:53513_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3
   [junit4]   2> 3759576 INFO  (jetty-launcher-10107-thread-3) [n:127.0.0.1:53513_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3
   [junit4]   2> 3759577 INFO  (jetty-launcher-10107-thread-3) [n:127.0.0.1:53513_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3
   [junit4]   2> 3759578 INFO  (jetty-launcher-10107-thread-3) [n:127.0.0.1:53513_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node3\.
   [junit4]   2> 3759629 INFO  (zkConnectionManagerCallback-10160-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3759630 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1007a003783000d, likely client has closed socket
   [junit4]   2> 3759634 INFO  (zkConnectionManagerCallback-10165-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3759635 INFO  (SUITE-AutoAddReplicasIntegrationTest-seed#[4AE7587C48CC21FA]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 3759637 INFO  (SUITE-AutoAddReplicasIntegrationTest-seed#[4AE7587C48CC21FA]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:53503/solr ready
   [junit4]   2> 3759669 INFO  (TEST-AutoAddReplicasIntegrationTest.testSimple-seed#[4AE7587C48CC21FA]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSimple
   [junit4]   2> 3759671 INFO  (qtp17263995-24483) [n:127.0.0.1:53508_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf&maxShardsPerNode=2&autoAddReplicas=true&name=testSimple1&nrtReplicas=2&action=CREATE&numShards=2&createNodeSet=127.0.0.1:53508_solr,127.0.0.1:53507_solr&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3759672 INFO  (OverseerThreadFactory-8366-thread-1-processing-n:127.0.0.1:53507_solr) [n:127.0.0.1:53507_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection testSimple1
   [junit4]   2> 3759782 DEBUG (OverseerThreadFactory-8366-thread-1-processing-n:127.0.0.1:53507_solr) [n:127.0.0.1:53507_solr    ] o.a.s.c.s.c.a.PolicyHelper Creating a new session
   [junit4]   2> 3759790 INFO  (qtp17263995-24466) [n:127.0.0.1:53508_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=6
   [junit4]   2> 3759796 INFO  (qtp897087-24478) [n:127.0.0.1:53513_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=3
   [junit4]   2> 3759815 INFO  (qtp16217251-24465) [n:127.0.0.1:53507_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=4
   [junit4]   2> 3759815 DEBUG (OverseerThreadFactory-8366-thread-1-processing-n:127.0.0.1:53507_solr) [n:127.0.0.1:53507_solr    ] o.a.s.c.s.c.a.PolicyHelper New session created
   [junit4]   2> 3759817 DEBUG (OverseerThreadFactory-8366-thread-1-processing-n:127.0.0.1:53507_solr) [n:127.0.0.1:53507_solr    ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 2046836340 sessionWrapper.createTime 2046836338384300, this.sessionWrapper.createTime 2046836338384300
   [junit4]   2> 3759822 INFO  (OverseerStateUpdate-72191735387848711-127.0.0.1:53507_solr-n_0000000000) [n:127.0.0.1:53507_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testSimple1_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:53507/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 3759824 INFO  (OverseerStateUpdate-72191735387848711-127.0.0.1:53507_solr-n_0000000000) [n:127.0.0.1:53507_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testSimple1_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:53508/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 3759827 INFO  (OverseerStateUpdate-72191735387848711-127.0.0.1:53507_solr-n_0000000000) [n:127.0.0.1:53507_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testSimple1_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:53508/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 3759828 INFO  (OverseerStateUpdate-72191735387848711-127.0.0.1:53507_solr-n_0000000000) [n:127.0.0.1:53507_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testSimple1_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:53507/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 3760032 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr    x:testSimple1_shard2_replica_n6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n6&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3760032 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr    x:testSimple1_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n2&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3760032 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr    x:testSimple1_shard2_replica_n4] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n4&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3760034 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr    x:testSimple1_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n1&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3760036 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr    x:testSimple1_shard2_replica_n4] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3760444 DEBUG (ScheduledTrigger-8357-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 3761080 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 3761080 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 3761080 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 3761094 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 3761096 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.s.IndexSchema [testSimple1_shard2_replica_n4] Schema name=minimal
   [junit4]   2> 3761097 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.IndexSchema [testSimple1_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 3761100 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.IndexSchema [testSimple1_shard2_replica_n6] Schema name=minimal
   [junit4]   2> 3761101 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.IndexSchema [testSimple1_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 3761103 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3761103 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard1_replica_n2' using configuration from collection testSimple1, trusted=true
   [junit4]   2> 3761103 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3761103 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard2_replica_n6' using configuration from collection testSimple1, trusted=true
   [junit4]   2> 3761104 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3761104 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard2_replica_n4' using configuration from collection testSimple1, trusted=true
   [junit4]   2> 3761104 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.testSimple1.shard1.replica_n2' (registry 'solr.core.testSimple1.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3
   [junit4]   2> 3761104 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.testSimple1.shard2.replica_n6' (registry 'solr.core.testSimple1.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3
   [junit4]   2> 3761104 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrCore [[testSimple1_shard1_replica_n2] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node1\testSimple1_shard1_replica_n2], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node1\.\testSimple1_shard1_replica_n2\data\]
   [junit4]   2> 3761104 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SolrCore [[testSimple1_shard2_replica_n6] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node2\testSimple1_shard2_replica_n6], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node2\.\testSimple1_shard2_replica_n6\data\]
   [junit4]   2> 3761105 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3761105 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard1_replica_n1' using configuration from collection testSimple1, trusted=true
   [junit4]   2> 3761104 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.testSimple1.shard2.replica_n4' (registry 'solr.core.testSimple1.shard2.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3
   [junit4]   2> 3761106 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SolrCore [[testSimple1_shard2_replica_n4] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node1\testSimple1_shard2_replica_n4], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node1\.\testSimple1_shard2_replica_n4\data\]
   [junit4]   2> 3761106 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.testSimple1.shard1.replica_n1' (registry 'solr.core.testSimple1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3
   [junit4]   2> 3761106 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrCore [[testSimple1_shard1_replica_n1] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node2\testSimple1_shard1_replica_n1], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node2\.\testSimple1_shard1_replica_n1\data\]
   [junit4]   2> 3761205 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3761205 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3761206 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3761206 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3761207 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3761207 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3761207 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3761207 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3761208 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3761208 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3761209 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.s.SolrIndexSearcher Opening [Searcher@1355d7b[testSimple1_shard2_replica_n4] main]
   [junit4]   2> 3761209 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3761209 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3761210 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 3761210 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@15da33f[testSimple1_shard1_replica_n1] main]
   [junit4]   2> 3761210 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 3761210 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3761211 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613838376547385344
   [junit4]   2> 3761211 INFO  (searcherExecutor-8373-thread-1-processing-n:127.0.0.1:53508_solr x:testSimple1_shard2_replica_n4 c:testSimple1 s:shard2 r:core_node7) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SolrCore [testSimple1_shard2_replica_n4] Registered new searcher Searcher@1355d7b[testSimple1_shard2_replica_n4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3761213 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 3761214 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 3761215 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3761216 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613838376552628224
   [junit4]   2> 3761216 INFO  (searcherExecutor-8374-thread-1-processing-n:127.0.0.1:53507_solr x:testSimple1_shard1_replica_n1 c:testSimple1 s:shard1 r:core_node3) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrCore [testSimple1_shard1_replica_n1] Registered new searcher Searcher@15da33f[testSimple1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3761217 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testSimple1/terms/shard2 to Terms{values={core_node7=0}, version=0}
   [junit4]   2> 3761222 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testSimple1/terms/shard1 to Terms{values={core_node3=0}, version=0}
   [junit4]   2> 3761222 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 3761225 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@1af9c96[testSimple1_shard1_replica_n2] main]
   [junit4]   2> 3761226 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 3761227 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 3761227 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 3761228 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3761228 INFO  (searcherExecutor-8371-thread-1-processing-n:127.0.0.1:53508_solr x:testSimple1_shard1_replica_n2 c:testSimple1 s:shard1 r:core_node5) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrCore [testSimple1_shard1_replica_n2] Registered new searcher Searcher@1af9c96[testSimple1_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3761229 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613838376566259712
   [junit4]   2> 3761232 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testSimple1/terms/shard1 to Terms{values={core_node3=0, core_node5=0}, version=1}
   [junit4]   2> 3761234 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3761234 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3761237 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3761237 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3761238 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.SolrIndexSearcher Opening [Searcher@1238660[testSimple1_shard2_replica_n6] main]
   [junit4]   2> 3761239 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 3761240 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 3761240 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3761241 INFO  (searcherExecutor-8372-thread-1-processing-n:127.0.0.1:53507_solr x:testSimple1_shard2_replica_n6 c:testSimple1 s:shard2 r:core_node8) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SolrCore [testSimple1_shard2_replica_n6] Registered new searcher Searcher@1238660[testSimple1_shard2_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3761241 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613838376578842624
   [junit4]   2> 3761244 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testSimple1/terms/shard2 to Terms{values={core_node7=0, core_node8=0}, version=1}
   [junit4]   2> 3761446 DEBUG (ScheduledTrigger-8357-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 3761723 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3761723 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3761723 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:53508/solr/testSimple1_shard2_replica_n4/
   [junit4]   2> 3761724 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.PeerSync PeerSync: core=testSimple1_shard2_replica_n4 url=http://127.0.0.1:53508/solr START replicas=[http://127.0.0.1:53507/solr/testSimple1_shard2_replica_n6/] nUpdates=100
   [junit4]   2> 3761724 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.PeerSync PeerSync: core=testSimple1_shard2_replica_n4 url=http://127.0.0.1:53508/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 3761725 INFO  (qtp16217251-24467) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.S.Request [testSimple1_shard2_replica_n6]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 3761727 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 3761727 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 3761727 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3761727 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3761727 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3761727 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:53507/solr/testSimple1_shard1_replica_n1/
   [junit4]   2> 3761727 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=testSimple1_shard1_replica_n1 url=http://127.0.0.1:53507/solr START replicas=[http://127.0.0.1:53508/solr/testSimple1_shard1_replica_n2/] nUpdates=100
   [junit4]   2> 3761728 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=testSimple1_shard1_replica_n1 url=http://127.0.0.1:53507/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 3761729 INFO  (qtp17263995-24471) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.S.Request [testSimple1_shard1_replica_n2]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 3761729 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 3761729 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 3761730 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3761730 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:53508/solr/testSimple1_shard2_replica_n4/ shard2
   [junit4]   2> 3761748 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:53507/solr/testSimple1_shard1_replica_n1/ shard1
   [junit4]   2> 3761851 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3761856 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n1&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1822
   [junit4]   2> 3761884 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3761890 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n4&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1857
   [junit4]   2> 3761996 INFO  (zkCallback-10129-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3761996 INFO  (zkCallback-10121-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3762262 INFO  (qtp17263995-24473) [n:127.0.0.1:53508_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n2&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2229
   [junit4]   2> 3762268 INFO  (qtp16217251-24485) [n:127.0.0.1:53507_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n6&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2236
   [junit4]   2> 3762268 DEBUG (OverseerThreadFactory-8366-thread-1-processing-n:127.0.0.1:53507_solr) [n:127.0.0.1:53507_solr    ] o.a.s.c.s.c.a.PolicyHelper session set to NULL
   [junit4]   2> 3762271 INFO  (qtp17263995-24483) [n:127.0.0.1:53508_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 3762368 INFO  (zkCallback-10129-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3762368 INFO  (zkCallback-10121-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3762460 DEBUG (ScheduledTrigger-8357-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 3763283 INFO  (qtp17263995-24483) [n:127.0.0.1:53508_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf&maxShardsPerNode=2&autoAddReplicas=true&name=testSimple1&nrtReplicas=2&action=CREATE&numShards=2&createNodeSet=127.0.0.1:53508_solr,127.0.0.1:53507_solr&wt=javabin&version=2} status=0 QTime=3612
   [junit4]   2> 3763284 INFO  (qtp17263995-24493) [n:127.0.0.1:53508_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf&maxShardsPerNode=2&autoAddReplicas=false&name=testSimple2&nrtReplicas=2&action=CREATE&numShards=2&createNodeSet=127.0.0.1:53507_solr,127.0.0.1:53513_solr&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3763290 INFO  (OverseerCollectionConfigSetProcessor-72191735387848711-127.0.0.1:53507_solr-n_0000000000) [n:127.0.0.1:53507_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3763323 INFO  (OverseerThreadFactory-8366-thread-2-processing-n:127.0.0.1:53507_solr) [n:127.0.0.1:53507_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection testSimple2
   [junit4]   2> 3763468 DEBUG (ScheduledTrigger-8357-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 3763533 DEBUG (OverseerThreadFactory-8366-thread-2-processing-n:127.0.0.1:53507_solr) [n:127.0.0.1:53507_solr    ] o.a.s.c.s.c.a.PolicyHelper Creating a new session
   [junit4]   2> 3763535 INFO  (qtp17263995-24488) [n:127.0.0.1:53508_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.testSimple1.shard2.replica_n4:INDEX.sizeInBytes&key=solr.core.testSimple1.shard1.replica_n2:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 3763539 INFO  (qtp17263995-24471) [n:127.0.0.1:53508_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=4
   [junit4]   2> 3763547 INFO  (qtp897087-24464) [n:127.0.0.1:53513_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=3
   [junit4]   2> 3763548 INFO  (qtp16217251-24468) [n:127.0.0.1:53507_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.testSimple1.shard2.replica_n6:INDEX.sizeInBytes&key=solr.core.testSimple1.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 3763551 INFO  (qtp16217251-24474) [n:127.0.0.1:53507_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=2
   [junit4]   2> 3763552 DEBUG (OverseerThreadFactory-8366-thread-2-processing-n:127.0.0.1:53507_solr) [n:127.0.0.1:53507_solr    ] o.a.s.c.s.c.a.PolicyHelper New session created
   [junit4]   2> 3763553 DEBUG (OverseerThreadFactory-8366-thread-2-processing-n:127.0.0.1:53507_solr) [n:127.0.0.1:53507_solr    ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 2046840075 sessionWrapper.createTime 2046840074746000, this.sessionWrapper.createTime 2046840074746000
   [junit4]   2> 3763555 INFO  (OverseerStateUpdate-72191735387848711-127.0.0.1:53507_solr-n_0000000000) [n:127.0.0.1:53507_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple2",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testSimple2_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:53513/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 3763557 INFO  (OverseerStateUpdate-72191735387848711-127.0.0.1:53507_solr-n_0000000000) [n:127.0.0.1:53507_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple2",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testSimple2_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:53513/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 3763559 INFO  (OverseerStateUpdate-72191735387848711-127.0.0.1:53507_solr-n_0000000000) [n:127.0.0.1:53507_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple2",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testSimple2_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:53513/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 3763561 INFO  (OverseerStateUpdate-72191735387848711-127.0.0.1:53507_solr-n_0000000000) [n:127.0.0.1:53507_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple2",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testSimple2_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:53507/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 3763766 INFO  (qtp16217251-24468) [n:127.0.0.1:53507_solr    x:testSimple2_shard2_replica_n6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=testSimple2_shard2_replica_n6&action=CREATE&numShards=2&collection=testSimple2&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3763767 INFO  (qtp897087-24479) [n:127.0.0.1:53513_solr    x:testSimple2_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testSimple2_shard1_replica_n1&action=CREATE&numShards=2&collection=testSimple2&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3763768 INFO  (qtp897087-24477) [n:127.0.0.1:53513_solr    x:testSimple2_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=testSimple2_shard1_replica_n2&action=CREATE&numShards=2&collection=testSimple2&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3763769 INFO  (qtp897087-24484) [n:127.0.0.1:53513_solr    x:testSimple2_shard2_replica_n4] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=testSimple2_shard2_replica_n4&action=CREATE&numShards=2&collection=testSimple2&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3764470 DEBUG (ScheduledTrigger-8357-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 3764834 INFO  (qtp16217251-24468) [n:127.0.0.1:53507_solr c:testSimple2 s:shard2 r:core_node8 x:testSimple2_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 3764834 INFO  (qtp897087-24484) [n:127.0.0.1:53513_solr c:testSimple2 s:shard2 r:core_node7 x:testSimple2_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 3764834 INFO  (qtp897087-24479) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 3764840 INFO  (qtp897087-24484) [n:127.0.0.1:53513_solr c:testSimple2 s:shard2 r:core_node7 x:testSimple2_shard2_replica_n4] o.a.s.s.IndexSchema [testSimple2_shard2_replica_n4] Schema name=minimal
   [junit4]   2> 3764840 INFO  (qtp16217251-24468) [n:127.0.0.1:53507_solr c:testSimple2 s:shard2 r:core_node8 x:testSimple2_shard2_replica_n6] o.a.s.s.IndexSchema [testSimple2_shard2_replica_n6] Schema name=minimal
   [junit4]   2> 3764842 INFO  (qtp897087-24479) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.s.IndexSchema [testSimple2_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 3764845 INFO  (qtp897087-24477) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node5 x:testSimple2_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 3764850 INFO  (qtp16217251-24468) [n:127.0.0.1:53507_solr c:testSimple2 s:shard2 r:core_node8 x:testSimple2_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3764850 INFO  (qtp897087-24484) [n:127.0.0.1:53513_solr c:testSimple2 s:shard2 r:core_node7 x:testSimple2_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3764850 INFO  (qtp16217251-24468) [n:127.0.0.1:53507_solr c:testSimple2 s:shard2 r:core_node8 x:testSimple2_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'testSimple2_shard2_replica_n6' using configuration from collection testSimple2, trusted=true
   [junit4]   2> 3764850 INFO  (qtp897087-24484) [n:127.0.0.1:53513_solr c:testSimple2 s:shard2 r:core_node7 x:testSimple2_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'testSimple2_shard2_replica_n4' using configuration from collection testSimple2, trusted=true
   [junit4]   2> 3764851 INFO  (qtp16217251-24468) [n:127.0.0.1:53507_solr c:testSimple2 s:shard2 r:core_node8 x:testSimple2_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.testSimple2.shard2.replica_n6' (registry 'solr.core.testSimple2.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3
   [junit4]   2> 3764851 INFO  (qtp16217251-24468) [n:127.0.0.1:53507_solr c:testSimple2 s:shard2 r:core_node8 x:testSimple2_shard2_replica_n6] o.a.s.c.SolrCore [[testSimple2_shard2_replica_n6] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node2\testSimple2_shard2_replica_n6], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node2\.\testSimple2_shard2_replica_n6\data\]
   [junit4]   2> 3764851 INFO  (qtp897087-24484) [n:127.0.0.1:53513_solr c:testSimple2 s:shard2 r:core_node7 x:testSimple2_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.testSimple2.shard2.replica_n4' (registry 'solr.core.testSimple2.shard2.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3
   [junit4]   2> 3764853 INFO  (qtp897087-24484) [n:127.0.0.1:53513_solr c:testSimple2 s:shard2 r:core_node7 x:testSimple2_shard2_replica_n4] o.a.s.c.SolrCore [[testSimple2_shard2_replica_n4] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node3\testSimple2_shard2_replica_n4], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node3\.\testSimple2_shard2_replica_n4\data\]
   [junit4]   2> 3764857 INFO  (qtp897087-24479) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3764857 INFO  (qtp897087-24479) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testSimple2_shard1_replica_n1' using configuration from collection testSimple2, trusted=true
   [junit4]   2> 3764858 INFO  (qtp897087-24479) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.testSimple2.shard1.replica_n1' (registry 'solr.core.testSimple2.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3
   [junit4]   2> 3764858 INFO  (qtp897087-24479) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.SolrCore [[testSimple2_shard1_replica_n1] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node3\testSimple2_shard1_replica_n1], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node3\.\testSimple2_shard1_replica_n1\data\]
   [junit4]   2> 3764859 INFO  (qtp897087-24477) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node5 x:testSimple2_shard1_replica_n2] o.a.s.s.IndexSchema [testSimple2_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 3764861 INFO  (qtp897087-24477) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node5 x:testSimple2_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3764861 INFO  (qtp897087-24477) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node5 x:testSimple2_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'testSimple2_shard1_replica_n2' using configuration from collection testSimple2, trusted=true
   [junit4]   2> 3764862 INFO  (qtp897087-24477) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node5 x:testSimple2_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.testSimple2.shard1.replica_n2' (registry 'solr.core.testSimple2.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1af7ee3
   [junit4]   2> 3764862 INFO  (qtp897087-24477) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node5 x:testSimple2_shard1_replica_n2] o.a.s.c.SolrCore [[testSimple2_shard1_replica_n2] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node3\testSimple2_shard1_replica_n2], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_4AE7587C48CC21FA-001\tempDir-001\node3\.\testSimple2_shard1_replica_n2\data\]
   [junit4]   2> 3765175 INFO  (qtp897087-24479) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3765175 INFO  (qtp897087-24479) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3765176 INFO  (qtp897087-24479) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3765176 INFO  (qtp897087-24479) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3765177 INFO  (qtp16217251-24468) [n:127.0.0.1:53507_solr c:testSimple2 s:shard2 r:core_node8 x:testSimple2_shard2_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3765177 INFO  (qtp16217251-24468) [n:127.0.0.1:53507_solr c:testSimple2 s:shard2 r:core_node8 x:testSimple2_shard2_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3765178 INFO  (qtp16217251-24468) [n:127.0.0.1:53507_solr c:testSimple2 s:shard2 r:core_node8 x:testSimple2_shard2_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3765178 INFO  (qtp16217251-24468) [n:127.0.0.1:53507_solr c:testSimple2 s:shard2 r:core_node8 x:testSimple2_shard2_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3765179 INFO  (qtp897087-24477) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node5 x:testSimple2_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3765179 INFO  (qtp897087-24477) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node5 x:testSimple2_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3765181 INFO  (qtp897087-24477) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node5 x:testSimple2_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3765181 INFO  (qtp897087-24477) [n:127.0.0.1:53513_solr c:testSimple2 s:shard1 r:core_node5 x:testSimple2_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3765181 INFO  (qtp897087-24484) [n:127.0.0.1:53513_solr c:testSimple2 s:shard2 r:core_node7 x:testSimple2_shard2_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3765181 INFO  (qtp897087-24484) [n:127.0.0.1:53513_solr c:testSimple2 s:shard2 r:core_node7 x:testSimple2_shard2_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3765187 INFO  (qtp897087-24484) [n:127.0.0.1:53513_solr c:testSimple2 s:shard2 r:core_node7 x:testSimple2_shard2_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3765187 INFO  (qtp897087-24484) [n:127.0.0.1:53513_solr c:testSimple2 s:shard2 r:core_node7 x:testSimple2_shard2_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit

[...truncated too long message...]

5)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   1> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
   [junit4]   1> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
   [junit4]   1> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
   [junit4]   1> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724)
   [junit4]   1> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]   1> at org.eclipse.jetty.server.Server.handle(Server.java:531)
   [junit4]   1> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
   [junit4]   1> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
   [junit4]   1> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
   [junit4]   1> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   1> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
   [junit4]   1> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
   [junit4]   1> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
   [junit4]   1> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
   [junit4]   1> at java.lang.Thread.run(Thread.java:748)
   [junit4]   1>
   [junit4]   1> -
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> no one is recoverying
   [junit4]   1> -
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> replica:core_node4 rstate:down live:true
   [junit4]   1> -
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> replica:core_node4 rstate:recovering live:true
   [junit4]   1> -
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> replica:core_node4 rstate:active live:true
   [junit4]   1> no one is recoverying
   [junit4]   1> -
   [junit4]   1> replica:core_node4 rstate:active live:true
   [junit4]   1> no one is recoverying
   [junit4]   1> 09:42:30.215 [TEST-CloudSolrClientTest.testRetryUpdatesWhenClusterStateIsStale-seed#[4A6696CFB3A8092]] ERROR org.apache.solr.client.solrj.impl.CloudSolrClient - Request to collection [stale_state_test_col] failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:50551/solr/stale_state_test_col_shard1_replica_n1: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]   1> <head>
   [junit4]   1> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
   [junit4]   1> <title>Error 404 Can not find: /solr/stale_state_test_col_shard1_replica_n1/update</title>
   [junit4]   1> </head>
   [junit4]   1> <body><h2>HTTP ERROR 404</h2>
   [junit4]   1> <p>Problem accessing /solr/stale_state_test_col_shard1_replica_n1/update. Reason:
   [junit4]   1> <pre>    Can not find: /solr/stale_state_test_col_shard1_replica_n1/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>
   [junit4]   1>
   [junit4]   1> </body>
   [junit4]   1> </html>
   [junit4]   1> , retry? 0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrClientTest -Dtests.method=preferLocalShardsTest -Dtests.seed=4A6696CFB3A8092 -Dtests.slow=true -Dtests.locale=ar-AE -Dtests.timezone=Europe/Budapest -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 6.18s J0 | CloudSolrClientTest.preferLocalShardsTest <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Response was not received from shards on a single node
   [junit4]    > at __randomizedtesting.SeedInfo.seed([4A6696CFB3A8092:F86BF1D505ED1768]:0)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClientTest.queryWithShardsPreferenceRules(CloudSolrClientTest.java:471)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClientTest.preferLocalShardsTest(CloudSolrClientTest.java:420)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   1> 09:42:47.926 [SUITE-CloudSolrClientTest-seed#[4A6696CFB3A8092]-worker] ERROR org.apache.zookeeper.server.ZooKeeperServer - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> NOTE: leaving temporary files on disk at: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-solrj\test\J0\temp\solr.client.solrj.impl.CloudSolrClientTest_4A6696CFB3A8092-001
   [junit4]   2> Oct 09, 2018 9:42:48 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=15072, maxDocsPerChunk=9, blockSize=4), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=15072, blockSize=4)), sim=RandomSimilarity(queryNorm=false): {}, locale=ar-AE, timezone=Europe/Budapest
   [junit4]   2> NOTE: Windows 10 10.0 x86/Oracle Corporation 1.8.0_172 (32-bit)/cpus=3,threads=1,free=70917264,total=306446336
   [junit4]   2> NOTE: All tests run in this JVM: [SchemaTest, SolrExampleJettyTest, TestPolicy, SolrParamTest, CeilingEvaluatorTest, MultiplyEvaluatorTest, TestCollectionAdminRequest, CommonAdminParamsTest, NotEvaluatorTest, LBHttpSolrClientBuilderTest, NormalDistributionEvaluatorTest, LargeVolumeEmbeddedTest, AppendEvaluatorTest, ExclusiveOrEvaluatorTest, TestFastInputStream, SolrDocumentTest, PowerEvaluatorTest, GraphTest, SolrZkClientTest, StreamingTest, TermsResponseTest, AnlysisResponseBaseTest, UuidEvaluatorTest, CumulativeProbabilityEvaluatorTest, SubtractEvaluatorTest, TestDelegationTokenResponse, ConcurrentUpdateSolrClientMultiCollectionTest, RecursiveEvaluatorTest, TestSuggesterResponse, TestCoreAdmin, TestLang, CosineEvaluatorTest, TestCollectionStateWatchers, TestV1toV2ApiMapper, ConcatOperationTest, SineEvaluatorTest, SolrExampleBinaryTest, SolrQueryTest, SolrSchemalessExampleTest, TestPolicy2, JettyWebappTest, LargeVolumeJettyTest, SolrExampleStreamingTest, CloudSolrClientTest]
   [junit4] Completed [107/157 (1!)] on J0 in 44.71s, 16 tests, 1 failure <<< FAILURES!

[...truncated 25105 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.8
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.8
[forbidden-apis] Reading bundled API signatures: jdk-non-portable
[forbidden-apis] Reading bundled API signatures: jdk-reflection
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.5
[forbidden-apis] Reading API signatures: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\tools\forbiddenApis\base.txt
[forbidden-apis] Reading API signatures: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\tools\forbiddenApis\servlet-api.txt
[forbidden-apis] Reading API signatures: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\tools\forbiddenApis\solr.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning classes for violations...
[forbidden-apis] Forbidden method invocation: java.io.InputStreamReader#<init>(java.io.InputStream) [Uses default charset]
[forbidden-apis]   in org.apache.solr.client.solrj.request.MultiContentWriterRequest (MultiContentWriterRequest.java:95)
[forbidden-apis] Scanned 1374 class file(s) for forbidden API invocations (in 2.20s), 1 error(s).

BUILD FAILED
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\build.xml:633: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\build.xml:117: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build.xml:381: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\common-build.xml:536: Check for forbidden API calls failed, see log.

Total time: 130 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-7.x-Windows (64bit/jdk-10.0.1) - Build # 828 - Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Windows/828/
Java: 64bit/jdk-10.0.1 -XX:-UseCompressedOops -XX:+UseParallelGC

27 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.backcompat.TestLuceneIndexBackCompat

Error Message:
Could not remove the following files (in the order of attempts):    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J0\temp\solr.backcompat.TestLuceneIndexBackCompat_6A53FA2B425D1B5E-001\7.5.0-nocfs-001\7.5.0-nocfs: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J0\temp\solr.backcompat.TestLuceneIndexBackCompat_6A53FA2B425D1B5E-001\7.5.0-nocfs-001\7.5.0-nocfs    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J0\temp\solr.backcompat.TestLuceneIndexBackCompat_6A53FA2B425D1B5E-001\7.5.0-nocfs-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J0\temp\solr.backcompat.TestLuceneIndexBackCompat_6A53FA2B425D1B5E-001\7.5.0-nocfs-001

Stack Trace:
java.io.IOException: Could not remove the following files (in the order of attempts):
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J0\temp\solr.backcompat.TestLuceneIndexBackCompat_6A53FA2B425D1B5E-001\7.5.0-nocfs-001\7.5.0-nocfs: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J0\temp\solr.backcompat.TestLuceneIndexBackCompat_6A53FA2B425D1B5E-001\7.5.0-nocfs-001\7.5.0-nocfs
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J0\temp\solr.backcompat.TestLuceneIndexBackCompat_6A53FA2B425D1B5E-001\7.5.0-nocfs-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J0\temp\solr.backcompat.TestLuceneIndexBackCompat_6A53FA2B425D1B5E-001\7.5.0-nocfs-001

        at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E]:0)
        at org.apache.lucene.util.IOUtils.rm(IOUtils.java:318)
        at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
        at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:57685/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:57685/solr
        at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E:ABA383876F0DD1F9]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:901)
        at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:566)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:58086/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:58086/solr
        at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E:ABA383876F0DD1F9]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:901)
        at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:566)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:58324/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:58324/solr
        at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E:ABA383876F0DD1F9]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:901)
        at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:566)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:58567/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:58567/solr
        at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E:ABA383876F0DD1F9]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:901)
        at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:566)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:57153/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:57153/solr
        at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E:ABA383876F0DD1F9]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:901)
        at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:566)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:52935/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:52935/solr
        at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E:ABA383876F0DD1F9]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:901)
        at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:566)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue.testDistributedQueueBlocking

Error Message:


Stack Trace:
java.util.concurrent.TimeoutException
        at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E:2FF98852060FA72A]:0)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204)
        at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.testDistributedQueueBlocking(TestSimDistributedQueue.java:102)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue.testDistributedQueue

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E]:0)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue.testDistributedQueueBlocking

Error Message:


Stack Trace:
java.util.concurrent.TimeoutException
        at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E:2FF98852060FA72A]:0)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204)
        at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.testDistributedQueueBlocking(TestSimDistributedQueue.java:117)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue.testDistributedQueue

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E]:0)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testNonRetryableRequests

Error Message:
expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
        at __randomizedtesting.SeedInfo.seed([F962D39C6207EF9A:518C140F9C4E9B96]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testNonRetryableRequests(CloudSolrClientTest.java:550)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testNonRetryableRequests

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
        at __randomizedtesting.SeedInfo.seed([F962D39C6207EF9A:518C140F9C4E9B96]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testNonRetryableRequests(CloudSolrClientTest.java:550)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testNonRetryableRequests

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
        at __randomizedtesting.SeedInfo.seed([F962D39C6207EF9A:518C140F9C4E9B96]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testNonRetryableRequests(CloudSolrClientTest.java:550)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testNonRetryableRequests

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
        at __randomizedtesting.SeedInfo.seed([F962D39C6207EF9A:518C140F9C4E9B96]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testNonRetryableRequests(CloudSolrClientTest.java:550)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testNonRetryableRequests

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
        at __randomizedtesting.SeedInfo.seed([F962D39C6207EF9A:518C140F9C4E9B96]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testNonRetryableRequests(CloudSolrClientTest.java:550)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:57487_solr, 127.0.0.1:57490_solr] Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/18)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node3":{           "core":"testSimple1_shard1_replica_n1",           "base_url":"http://127.0.0.1:57487/solr",           "node_name":"127.0.0.1:57487_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node12":{           "core":"testSimple1_shard1_replica_n11",           "base_url":"http://127.0.0.1:57487/solr",           "node_name":"127.0.0.1:57487_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node8":{           "core":"testSimple1_shard2_replica_n6",           "base_url":"http://127.0.0.1:57487/solr",           "node_name":"127.0.0.1:57487_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node10":{           "core":"testSimple1_shard2_replica_n9",           "base_url":"http://127.0.0.1:57497/solr",           "node_name":"127.0.0.1:57497_solr",           "state":"down",           "type":"NRT",           "force_set_state":"false"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:57487_solr, 127.0.0.1:57490_solr]
Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/18)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "core":"testSimple1_shard1_replica_n1",
          "base_url":"http://127.0.0.1:57487/solr",
          "node_name":"127.0.0.1:57487_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node12":{
          "core":"testSimple1_shard1_replica_n11",
          "base_url":"http://127.0.0.1:57487/solr",
          "node_name":"127.0.0.1:57487_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node8":{
          "core":"testSimple1_shard2_replica_n6",
          "base_url":"http://127.0.0.1:57487/solr",
          "node_name":"127.0.0.1:57487_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node10":{
          "core":"testSimple1_shard2_replica_n9",
          "base_url":"http://127.0.0.1:57497/solr",
          "node_name":"127.0.0.1:57497_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E:52E0DED565AECF8F]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:57586_solr, 127.0.0.1:57587_solr] Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/19)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node3":{           "core":"testSimple1_shard1_replica_n1",           "base_url":"http://127.0.0.1:57587/solr",           "node_name":"127.0.0.1:57587_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node12":{           "core":"testSimple1_shard1_replica_n11",           "base_url":"http://127.0.0.1:57587/solr",           "node_name":"127.0.0.1:57587_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node8":{           "core":"testSimple1_shard2_replica_n6",           "base_url":"http://127.0.0.1:57587/solr",           "node_name":"127.0.0.1:57587_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node10":{           "core":"testSimple1_shard2_replica_n9",           "base_url":"http://127.0.0.1:57592/solr",           "node_name":"127.0.0.1:57592_solr",           "state":"down",           "type":"NRT",           "force_set_state":"false"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:57586_solr, 127.0.0.1:57587_solr]
Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/19)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "core":"testSimple1_shard1_replica_n1",
          "base_url":"http://127.0.0.1:57587/solr",
          "node_name":"127.0.0.1:57587_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node12":{
          "core":"testSimple1_shard1_replica_n11",
          "base_url":"http://127.0.0.1:57587/solr",
          "node_name":"127.0.0.1:57587_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node8":{
          "core":"testSimple1_shard2_replica_n6",
          "base_url":"http://127.0.0.1:57587/solr",
          "node_name":"127.0.0.1:57587_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node10":{
          "core":"testSimple1_shard2_replica_n9",
          "base_url":"http://127.0.0.1:57592/solr",
          "node_name":"127.0.0.1:57592_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E:52E0DED565AECF8F]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple

Error Message:
Error from server at http://127.0.0.1:57925/solr: create the collection time out:180s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:57925/solr: create the collection time out:180s
        at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E:52E0DED565AECF8F]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:72)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:57994_solr, 127.0.0.1:57996_solr] Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/18)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node3":{           "core":"testSimple1_shard1_replica_n1",           "base_url":"http://127.0.0.1:57996/solr",           "node_name":"127.0.0.1:57996_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node12":{           "core":"testSimple1_shard1_replica_n11",           "base_url":"http://127.0.0.1:57992/solr",           "node_name":"127.0.0.1:57992_solr",           "state":"down",           "type":"NRT",           "force_set_state":"false"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node8":{           "core":"testSimple1_shard2_replica_n6",           "base_url":"http://127.0.0.1:57996/solr",           "node_name":"127.0.0.1:57996_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node10":{           "core":"testSimple1_shard2_replica_n9",           "base_url":"http://127.0.0.1:57996/solr",           "node_name":"127.0.0.1:57996_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:57994_solr, 127.0.0.1:57996_solr]
Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/18)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "core":"testSimple1_shard1_replica_n1",
          "base_url":"http://127.0.0.1:57996/solr",
          "node_name":"127.0.0.1:57996_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node12":{
          "core":"testSimple1_shard1_replica_n11",
          "base_url":"http://127.0.0.1:57992/solr",
          "node_name":"127.0.0.1:57992_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node8":{
          "core":"testSimple1_shard2_replica_n6",
          "base_url":"http://127.0.0.1:57996/solr",
          "node_name":"127.0.0.1:57996_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node10":{
          "core":"testSimple1_shard2_replica_n9",
          "base_url":"http://127.0.0.1:57996/solr",
          "node_name":"127.0.0.1:57996_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E:52E0DED565AECF8F]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:57393_solr, 127.0.0.1:57402_solr] Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/17)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node6":{           "core":"testSimple1_shard1_replica_n2",           "base_url":"http://127.0.0.1:57402/solr",           "node_name":"127.0.0.1:57402_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node12":{           "core":"testSimple1_shard1_replica_n11",           "base_url":"http://127.0.0.1:57392/solr",           "node_name":"127.0.0.1:57392_solr",           "state":"down",           "type":"NRT",    

[...truncated too long message...]

stRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at app//org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
   [junit4]   2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
   [junit4]   2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
   [junit4]   2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
   [junit4]   2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>         at app//org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at app//org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at app//org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at app//org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at app//org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at app//org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
   [junit4]   2> Okt 10, 2018 11:00:27 SUBBAAHI com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=49, name=sdqtest--1-thread-1, state=TIMED_WAITING, group=TGRP-TestSimGenericDistributedQueue]
   [junit4]   2>    2) Thread[id=14, name=TEST-TestSimGenericDistributedQueue.testDistributedQueue-seed#[6A53FA2B425D1B5E], state=TIMED_WAITING, group=TGRP-TestSimGenericDistributedQueue]
   [junit4]   2> 7282794 INFO  (TEST-TestSimGenericDistributedQueue.testDistributedQueue-seed#[6A53FA2B425D1B5E]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDistributedQueue
   [junit4]   2> NOTE: leaving temporary files on disk at: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue_6A53FA2B425D1B5E-001
   [junit4]   2> Oct 10, 2018 2:00:27 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=269, maxMBSortInHeap=5.556022281545893, sim=RandomSimilarity(queryNorm=true): {}, locale=twq, timezone=Etc/GMT-9
   [junit4]   2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 10.0.1 (64-bit)/cpus=3,threads=1,free=93947320,total=171966464
   [junit4]   2> NOTE: All tests run in this JVM: [TestSimGenericDistributedQueue]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSimGenericDistributedQueue -Dtests.seed=6A53FA2B425D1B5E -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=twq -Dtests.timezone=Etc/GMT-9 -Dtests.asserts=true -Dtests.file.encoding=Cp1252
   [junit4] ERROR   0.00s J0 | TestSimGenericDistributedQueue (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > at __randomizedtesting.SeedInfo.seed([6A53FA2B425D1B5E]:0)
   [junit4] Completed [19/20 (11!)] on J0 in 7281.66s, 2 tests, 3 errors <<< FAILURES!
   [junit4]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue
   [junit4] OK      2.11s J0 | TestSimGenericDistributedQueue.testDistributedQueueBlocking
   [junit4] OK      2.00s J0 | TestSimGenericDistributedQueue.testDistributedQueue
   [junit4] OK      0.23s J0 | TestSimGenericDistributedQueue.testLocallyOffer
   [junit4] OK      2.01s J0 | TestSimGenericDistributedQueue.testDistributedQueue {#2}
   [junit4] OK      1.51s J0 | TestSimGenericDistributedQueue.testPeekElements
   [junit4] Completed [20/20 (11!)] on J0 in 7.88s, 5 tests
   [junit4]
   [junit4]
   [junit4] Tests with failures [seed: 6A53FA2B425D1B5E] (first 10 out of 13):
   [junit4]   - org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue.testDistributedQueueBlocking
   [junit4]   - org.apache.solr.cloud.TestTlogReplica.testRecovery
   [junit4]   - org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple
   [junit4]   - org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple
   [junit4]   - org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple
   [junit4]   - org.apache.solr.cloud.TestTlogReplica.testRecovery
   [junit4]   - org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple
   [junit4]   - org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple
   [junit4]   - org.apache.solr.cloud.TestTlogReplica.testRecovery
   [junit4]   - org.apache.solr.cloud.TestTlogReplica.testRecovery
   [junit4]
   [junit4]
   [junit4] JVM J0:     0.81 ..  7291.41 =  7290.60s
   [junit4] JVM J1:     1.06 ..  1450.58 =  1449.52s
   [junit4] Execution time total: 2 hours 1 minute 31 seconds
   [junit4] Tests summary: 20 suites, 97 tests, 1 suite-level error, 3 errors, 9 failures

BUILD FAILED
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\common-build.xml:1567: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\common-build.xml:1092: There were test failures: 20 suites, 97 tests, 1 suite-level error, 3 errors, 9 failures [seed: 6A53FA2B425D1B5E]

Total time: 121 minutes 34 seconds

[repro] Setting last failure code to 1

[repro] Failures:
[repro]   0/5 failed: org.apache.solr.backcompat.TestLuceneIndexBackCompat
[repro]   1/5 failed: org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue
[repro]   5/5 failed: org.apache.solr.client.solrj.impl.CloudSolrClientTest
[repro]   5/5 failed: org.apache.solr.cloud.TestTlogReplica
[repro]   5/5 failed: org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest
[repro] Exiting with code 1

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>IF EXIST lucene\build move /y lucene\build lucene\build.repro
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>IF EXIST solr\build move /y solr\build solr\build.repro
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>IF EXIST lucene\build.orig move /y lucene\build.orig lucene\build
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>IF EXIST solr\build.orig move /y solr\build.orig solr\build
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>exit 0
Archiving artifacts
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=2aed752f9e76a867c328ef04eaae8326f9fb6d69, workspace=C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows
[WARNINGS] Computing warning deltas based on reference build #826
Recording test results
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-7.x-Windows (64bit/jdk-10.0.1) - Build # 829 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Windows/829/
Java: 64bit/jdk-10.0.1 -XX:+UseCompressedOops -XX:+UseParallelGC

11 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testEventQueue

Error Message:
action wasn't interrupted

Stack Trace:
java.lang.AssertionError: action wasn't interrupted
        at __randomizedtesting.SeedInfo.seed([1180252F246882B2:D83567812D0F4447]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testEventQueue(TestSimTriggerIntegration.java:683)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  junit.framework.TestSuite.org.apache.solr.rest.TestManagedResourceStorage

Error Message:
SolrCore.getOpenCount()==2

Stack Trace:
java.lang.RuntimeException: SolrCore.getOpenCount()==2
        at __randomizedtesting.SeedInfo.seed([1180252F246882B2]:0)
        at org.apache.solr.util.TestHarness.close(TestHarness.java:380)
        at org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:802)
        at org.apache.solr.cloud.AbstractZkTestCase.azt_afterClass(AbstractZkTestCase.java:147)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  junit.framework.TestSuite.org.apache.solr.rest.TestManagedResourceStorage

Error Message:
Could not remove the following files (in the order of attempts):    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2\log.1: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2\log.1: The process cannot access the file because it is being used by another process.     C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001

Stack Trace:
java.io.IOException: Could not remove the following files (in the order of attempts):
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2\log.1: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2\log.1: The process cannot access the file because it is being used by another process.

   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001

        at org.apache.lucene.util.IOUtils.rm(IOUtils.java:318)
        at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
        at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  junit.framework.TestSuite.org.apache.solr.rest.TestManagedResourceStorage

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.rest.TestManagedResourceStorage:     1) Thread[id=50, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestManagedResourceStorage]         at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)         at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)         at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)         at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)         at java.base@10.0.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:89)         at java.base@10.0.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:100)         at app//org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196)         at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)    2) Thread[id=51, name=SessionTracker, state=TIMED_WAITING, group=TGRP-TestManagedResourceStorage]         at java.base@10.0.1/java.lang.Object.wait(Native Method)         at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:147)    3) Thread[id=49, name=Thread-0, state=WAITING, group=TGRP-TestManagedResourceStorage]         at java.base@10.0.1/java.lang.Object.wait(Native Method)         at java.base@10.0.1/java.lang.Thread.join(Thread.java:1353)         at java.base@10.0.1/java.lang.Thread.join(Thread.java:1427)         at app//org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:313)         at app//org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:313)         at app//org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:496)    4) Thread[id=53, name=ProcessThread(sid:0 cport:53355):, state=WAITING, group=TGRP-TestManagedResourceStorage]         at java.base@10.0.1/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@10.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@10.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2075)         at java.base@10.0.1/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)         at app//org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:123)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.rest.TestManagedResourceStorage:
   1) Thread[id=50, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestManagedResourceStorage]
        at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at java.base@10.0.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:89)
        at java.base@10.0.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:100)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196)
        at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
   2) Thread[id=51, name=SessionTracker, state=TIMED_WAITING, group=TGRP-TestManagedResourceStorage]
        at java.base@10.0.1/java.lang.Object.wait(Native Method)
        at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:147)
   3) Thread[id=49, name=Thread-0, state=WAITING, group=TGRP-TestManagedResourceStorage]
        at java.base@10.0.1/java.lang.Object.wait(Native Method)
        at java.base@10.0.1/java.lang.Thread.join(Thread.java:1353)
        at java.base@10.0.1/java.lang.Thread.join(Thread.java:1427)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:313)
        at app//org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:313)
        at app//org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:496)
   4) Thread[id=53, name=ProcessThread(sid:0 cport:53355):, state=WAITING, group=TGRP-TestManagedResourceStorage]
        at java.base@10.0.1/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@10.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@10.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2075)
        at java.base@10.0.1/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
        at app//org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:123)
        at __randomizedtesting.SeedInfo.seed([1180252F246882B2]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.rest.TestManagedResourceStorage

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=50, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestManagedResourceStorage]         at java.base@10.0.1/sun.nio.ch.Util$2.toArray(Util.java:342)         at java.base@10.0.1/java.util.ArrayList.<init>(ArrayList.java:179)         at app//org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:201)         at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=50, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestManagedResourceStorage]
        at java.base@10.0.1/sun.nio.ch.Util$2.toArray(Util.java:342)
        at java.base@10.0.1/java.util.ArrayList.<init>(ArrayList.java:179)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:201)
        at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
        at __randomizedtesting.SeedInfo.seed([1180252F246882B2]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.rest.TestManagedResourceStorage

Error Message:
SolrCore.getOpenCount()==1

Stack Trace:
java.lang.RuntimeException: SolrCore.getOpenCount()==1
        at __randomizedtesting.SeedInfo.seed([1180252F246882B2]:0)
        at org.apache.solr.util.TestHarness.close(TestHarness.java:380)
        at org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:802)
        at org.apache.solr.cloud.AbstractZkTestCase.azt_afterClass(AbstractZkTestCase.java:147)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  junit.framework.TestSuite.org.apache.solr.rest.TestManagedResourceStorage

Error Message:
Could not remove the following files (in the order of attempts):    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2\log.1: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2\log.1: The process cannot access the file because it is being used by another process.     C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001

Stack Trace:
java.io.IOException: Could not remove the following files (in the order of attempts):
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2\log.1: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2\log.1: The process cannot access the file because it is being used by another process.

   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001

        at org.apache.lucene.util.IOUtils.rm(IOUtils.java:318)
        at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
        at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  junit.framework.TestSuite.org.apache.solr.rest.TestManagedResourceStorage

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.rest.TestManagedResourceStorage:     1) Thread[id=36287, name=Thread-6689, state=WAITING, group=TGRP-TestManagedResourceStorage]         at java.base@10.0.1/java.lang.Object.wait(Native Method)         at java.base@10.0.1/java.lang.Thread.join(Thread.java:1353)         at java.base@10.0.1/java.lang.Thread.join(Thread.java:1427)         at app//org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:313)         at app//org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:313)         at app//org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:496)    2) Thread[id=36288, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestManagedResourceStorage]         at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)         at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)         at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)         at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)         at java.base@10.0.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:89)         at java.base@10.0.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:100)         at app//org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196)         at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)    3) Thread[id=36291, name=ProcessThread(sid:0 cport:61382):, state=WAITING, group=TGRP-TestManagedResourceStorage]         at java.base@10.0.1/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@10.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@10.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2075)         at java.base@10.0.1/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)         at app//org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:123)    4) Thread[id=36289, name=SessionTracker, state=TIMED_WAITING, group=TGRP-TestManagedResourceStorage]         at java.base@10.0.1/java.lang.Object.wait(Native Method)         at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:147)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.rest.TestManagedResourceStorage:
   1) Thread[id=36287, name=Thread-6689, state=WAITING, group=TGRP-TestManagedResourceStorage]
        at java.base@10.0.1/java.lang.Object.wait(Native Method)
        at java.base@10.0.1/java.lang.Thread.join(Thread.java:1353)
        at java.base@10.0.1/java.lang.Thread.join(Thread.java:1427)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:313)
        at app//org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:313)
        at app//org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:496)
   2) Thread[id=36288, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestManagedResourceStorage]
        at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at java.base@10.0.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:89)
        at java.base@10.0.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:100)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196)
        at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
   3) Thread[id=36291, name=ProcessThread(sid:0 cport:61382):, state=WAITING, group=TGRP-TestManagedResourceStorage]
        at java.base@10.0.1/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@10.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@10.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2075)
        at java.base@10.0.1/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
        at app//org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:123)
   4) Thread[id=36289, name=SessionTracker, state=TIMED_WAITING, group=TGRP-TestManagedResourceStorage]
        at java.base@10.0.1/java.lang.Object.wait(Native Method)
        at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:147)
        at __randomizedtesting.SeedInfo.seed([1180252F246882B2]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.rest.TestManagedResourceStorage

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=36288, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestManagedResourceStorage]         at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl.resetWakeupSocket0(Native Method)         at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl.resetWakeupSocket(WindowsSelectorImpl.java:474)         at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:174)         at java.base@10.0.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:89)         at java.base@10.0.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:100)         at app//org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196)         at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=36288, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestManagedResourceStorage]
        at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl.resetWakeupSocket0(Native Method)
        at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl.resetWakeupSocket(WindowsSelectorImpl.java:474)
        at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:174)
        at java.base@10.0.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:89)
        at java.base@10.0.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:100)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196)
        at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
        at __randomizedtesting.SeedInfo.seed([1180252F246882B2]:0)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue.testDistributedQueueBlocking

Error Message:


Stack Trace:
java.util.concurrent.TimeoutException
        at __randomizedtesting.SeedInfo.seed([1180252F246882B2:542A5756603A3EC6]:0)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204)
        at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.testDistributedQueueBlocking(TestSimDistributedQueue.java:102)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue:     1) Thread[id=17231, name=sdqtest--5298-thread-1, state=TIMED_WAITING, group=TGRP-TestSimGenericDistributedQueue]         at java.base@10.0.1/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@10.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@10.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2117)         at app//org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:194)         at app//org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:167)         at app//org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.lambda$testDistributedQueueBlocking$0(TestSimDistributedQueue.java:93)         at app//org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue$$Lambda$1268/218284661.call(Unknown Source)         at java.base@10.0.1/java.util.concurrent.FutureTask.run(FutureTask.java:264)         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$270/398953871.run(Unknown Source)         at java.base@10.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)         at java.base@10.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)         at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue:
   1) Thread[id=17231, name=sdqtest--5298-thread-1, state=TIMED_WAITING, group=TGRP-TestSimGenericDistributedQueue]
        at java.base@10.0.1/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@10.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@10.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2117)
        at app//org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:194)
        at app//org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:167)
        at app//org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.lambda$testDistributedQueueBlocking$0(TestSimDistributedQueue.java:93)
        at app//org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue$$Lambda$1268/218284661.call(Unknown Source)
        at java.base@10.0.1/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$270/398953871.run(Unknown Source)
        at java.base@10.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
        at java.base@10.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
        at __randomizedtesting.SeedInfo.seed([1180252F246882B2]:0)




Build Log:
[...truncated 13469 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue
   [junit4]   2> Creating dataDir: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue_1180252F246882B2-001\init-core-data-001
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSimGenericDistributedQueue -Dtests.method=testDistributedQueueBlocking -Dtests.seed=1180252F246882B2 -Dtests.slow=true -Dtests.locale=en-GH -Dtests.timezone=America/Bogota -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   2.00s J0 | TestSimGenericDistributedQueue.testDistributedQueueBlocking <<<
   [junit4]    > Throwable #1: java.util.concurrent.TimeoutException
   [junit4]    > at __randomizedtesting.SeedInfo.seed([1180252F246882B2:542A5756603A3EC6]:0)
   [junit4]    > at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204)
   [junit4]    > at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.testDistributedQueueBlocking(TestSimDistributedQueue.java:102)
   [junit4]    > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   [junit4]    > at java.base/java.lang.Thread.run(Thread.java:844)
   [junit4]   2> NOTE: leaving temporary files on disk at: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue_1180252F246882B2-001
   [junit4]   2> Oct 10, 2018 2:06:54 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> Oct 10, 2018 2:08:15 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue:
   [junit4]   2>    1) Thread[id=17231, name=sdqtest--5298-thread-1, state=TIMED_WAITING, group=TGRP-TestSimGenericDistributedQueue]
   [junit4]   2>         at java.base@10.0.1/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@10.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@10.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2117)
   [junit4]   2>         at app//org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:194)
   [junit4]   2>         at app//org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:167)
   [junit4]   2>         at app//org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.lambda$testDistributedQueueBlocking$0(TestSimDistributedQueue.java:93)
   [junit4]   2>         at app//org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue$$Lambda$1268/218284661.call(Unknown Source)
   [junit4]   2>         at java.base@10.0.1/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   [junit4]   2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
   [junit4]   2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$270/398953871.run(Unknown Source)
   [junit4]   2>         at java.base@10.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
   [junit4]   2>         at java.base@10.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
   [junit4]   2>         at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
   [junit4]   2> Oct 10, 2018 2:08:15 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=17231, name=sdqtest--5298-thread-1, state=TIMED_WAITING, group=TGRP-TestSimGenericDistributedQueue]
   [junit4]   2> Oct 10, 2018 2:08:15 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=313, maxMBSortInHeap=6.2454512598856065, sim=RandomSimilarity(queryNorm=true): {}, locale=en-GH, timezone=America/Bogota
   [junit4]   2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 10.0.1 (64-bit)/cpus=3,threads=1,free=155821536,total=434110464
   [junit4]   2> NOTE: All tests run in this JVM: [SolrCoreTest, TestUtilizeNode, AnalyticsMergeStrategyTest, IndexSchemaTest, FastVectorHighlighterTest, TestPivotHelperCode, DirectSolrSpellCheckerTest, MaxSizeAutoCommitTest, TestBlobHandler, TestSimNodeAddedTrigger, TestRestManager, SOLR749Test, TestMultiWordSynonyms, HdfsBasicDistributedZkTest, TestLegacyNumericUtils, SolrRrdBackendFactoryTest, TestStressLiveNodes, ZkNodePropsTest, LeaderTragicEventTest, TestSolrCoreParser, TestReplicationHandlerBackup, TestFileDictionaryLookup, TestScoreJoinQPNoScore, BlockJoinFacetDistribTest, HdfsRecoverLeaseTest, SliceStateTest, TestSimExecutePlanAction, TestCloudRecovery, TestPullReplica, CdcrBootstrapTest, SolrCloudReportersTest, TestStressReorder, TestLockTree, TestEmbeddedSolrServerConstructors, LeaderFailoverAfterPartitionTest, OutputWriterTest, SearchHandlerTest, MultiThreadedOCPTest, VMParamsZkACLAndCredentialsProvidersTest, TestZkChroot, TestDFRSimilarityFactory, TestCSVResponseWriter, TestLeaderElectionZkExpiry, BooleanFieldTest, FacetPivotSmallTest, NodeAddedTriggerTest, ScriptEngineTest, SystemLogListenerTest, EmbeddedSolrNoSerializeTest, ResponseBuilderTest, TestImplicitCoreProperties, PhrasesIdentificationComponentTest, SystemInfoHandlerTest, TestSimpleQParserPlugin, HLLSerializationTest, TestFuzzyAnalyzedSuggestions, WrapperMergePolicyFactoryTest, CopyFieldTest, TestNumericTerms64, XmlUpdateRequestHandlerTest, CachingDirectoryFactoryTest, ZkSolrClientTest, MoveReplicaHDFSFailoverTest, TestHashPartitioner, HdfsUnloadDistributedZkTest, TestExpandComponent, BinaryUpdateRequestHandlerTest, PeerSyncWithBufferUpdatesTest, RulesTest, CoreAdminOperationTest, DistanceFunctionTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestSolrCLIRunExample, TestCloudNestedDocsSort, ReplaceNodeNoTargetTest, QueryResultKeyTest, TestWordDelimiterFilterFactory, DirectUpdateHandlerTest, TestFieldTypeResource, TestConfig, BigEndianAscendingWordSerializerTest, SaslZkACLProviderTest, HdfsRestartWhileUpdatingTest, SimpleMLTQParserTest, BadCopyFieldTest, RollingRestartTest, TestGeoJSONResponseWriter, TestBadConfig, TestSolrJ, TestRestoreCore, AliasIntegrationTest, TestRTGBase, TestUniqueKeyFieldResource, TestRawTransformer, CreateRoutedAliasTest, TestOrdValues, FieldMutatingUpdateProcessorTest, DeleteReplicaTest, TestSolr4Spatial, DistributedFacetSimpleRefinementLongTailTest, TestCrossCoreJoin, TestDistributedGrouping, TestDynamicFieldResource, SolrGraphiteReporterTest, TestRebalanceLeaders, DistributedQueryElevationComponentTest, TestJsonFacetsWithNestedObjects, OpenCloseCoreStressTest, TestOmitPositions, TestRandomFlRTGCloud, TestShortCircuitedRequests, MultiTermTest, SuggesterTSTTest, TestCollationField, TestFieldCacheVsDocValues, SolrInfoBeanTest, TestInitParams, TestManagedSynonymFilterFactory, SchemaVersionSpecificBehaviorTest, TestDownShardTolerantSearch, TestFieldCacheSort, AtomicUpdatesTest, Tagger2Test, UpdateRequestProcessorFactoryTest, TestSchemaManager, TransactionLogTest, TestDistributedSearch, TestHdfsUpdateLog, TestSSLRandomization, TestXmlQParser, TestStreamBody, LIRRollingUpdatesTest, TestCustomSort, PolyFieldTest, AnalysisAfterCoreReloadTest, ClusterStateUpdateTest, TestFilteredDocIdSet, TemplateUpdateProcessorTest, TestSortableTextField, TestPayloadScoreQParserPlugin, TestSimDistribStateManager, CollectionStateFormat2Test, TestDistribIDF, TestConfigSetsAPIExclusivity, CoreAdminCreateDiscoverTest, CoreMergeIndexesAdminHandlerTest, TestAnalyzedSuggestions, TestJoin, LoggingHandlerTest, ShowFileRequestHandlerTest, DocumentAnalysisRequestHandlerTest, PrimUtilsTest, TestCollectionsAPIViaSolrCloudCluster, TestTestInjection, NumberUtilsTest, TestLuceneMatchVersion, NumericFieldsTest, SoftAutoCommitTest, DistributedExpandComponentTest, TestCoreDiscovery, TestRandomFaceting, TestSchemaVersionResource, TriggerIntegrationTest, MetricsConfigTest, OutOfBoxZkACLAndCredentialsProvidersTest, TestSimGenericDistributedQueue]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSimGenericDistributedQueue -Dtests.seed=1180252F246882B2 -Dtests.slow=true -Dtests.locale=en-GH -Dtests.timezone=America/Bogota -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | TestSimGenericDistributedQueue (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue:
   [junit4]    >    1) Thread[id=17231, name=sdqtest--5298-thread-1, state=TIMED_WAITING, group=TGRP-TestSimGenericDistributedQueue]
   [junit4]    >         at java.base@10.0.1/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@10.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@10.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2117)
   [junit4]    >         at app//org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:194)
   [junit4]    >         at app//org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:167)
   [junit4]    >         at app//org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.lambda$testDistributedQueueBlocking$0(TestSimDistributedQueue.java:93)
   [junit4]    >         at app//org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue$$Lambda$1268/218284661.call(Unknown Source)
   [junit4]    >         at java.base@10.0.1/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   [junit4]    >         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
   [junit4]    >         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$270/398953871.run(Unknown Source)
   [junit4]    >         at java.base@10.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
   [junit4]    >         at java.base@10.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
   [junit4]    >         at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([1180252F246882B2]:0)
   [junit4] Completed [283/834 (1!)] on J0 in 87.98s, 5 tests, 2 errors <<< FAILURES!

[...truncated 940 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
   [junit4]   2> Creating dataDir: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.autoscaling.sim.TestSimTriggerIntegration_1180252F246882B2-001\init-core-data-001
   [junit4]   2> 3073155 DEBUG (SUITE-TestSimTriggerIntegration-seed#[1180252F246882B2]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10020_solr
   [junit4]   2> 3073155 DEBUG (SUITE-TestSimTriggerIntegration-seed#[1180252F246882B2]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=0
   [junit4]   2> 3073155 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 3073155 DEBUG (SUITE-TestSimTriggerIntegration-seed#[1180252F246882B2]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
   [junit4]   2> 3073155 INFO  (SUITE-TestSimTriggerIntegration-seed#[1180252F246882B2]-worker) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3073155 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
   [junit4]   2> 3073155 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
   [junit4]   2> 3073155 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
   [junit4]   2> 3073156 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10021_solr, 127.0.0.1:10020_solr]
   [junit4]   2> 3073156 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3073156 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
   [junit4] IGNOR/A 0.00s J0 | TestSimTriggerIntegration.testNodeAddedTrigger
   [junit4]    > Assumption #1: 'badapple' test group is disabled (@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028"))
   [junit4] IGNOR/A 0.00s J0 | TestSimTriggerIntegration.testNodeAddedTriggerRestoreState
   [junit4]    > Assumption #1: 'badapple' test group is disabled (@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028"))
   [junit4]   2> 3073156 DEBUG (ScheduledTrigger-13297-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4] IGNOR/A 0.00s J0 | TestSimTriggerIntegration.testCooldown
   [junit4]    > Assumption #1: 'badapple' test group is disabled (@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028"))
   [junit4]   2> 3073159 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 3073159 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 3073159 INFO  (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 3073160 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 3073160 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 3073160 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3073160 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 3073160 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 3073160 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 3073160 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1259) ~[?:?]
   [junit4]   2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:317) ~[?:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:177) [java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:844) [?:?]
   [junit4]   2> 3073160 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 3073160 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 3073161 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 3073161 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 3073161 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
   [junit4]   2> 3073161 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 3073161 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10021_solr, 127.0.0.1:10020_solr]
   [junit4]   2> 3073162 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3073162 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 3073162 DEBUG (ScheduledTrigger-13302-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073182 DEBUG (ScheduledTrigger-13302-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073204 DEBUG (ScheduledTrigger-13302-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073225 DEBUG (ScheduledTrigger-13302-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073246 DEBUG (ScheduledTrigger-13302-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073266 DEBUG (ScheduledTrigger-13302-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073288 DEBUG (ScheduledTrigger-13302-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073309 DEBUG (ScheduledTrigger-13302-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073330 DEBUG (ScheduledTrigger-13302-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073351 DEBUG (ScheduledTrigger-13302-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073361 DEBUG (simCloudManagerPool-13301-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=1
   [junit4]   2> 3073362 DEBUG (simCloudManagerPool-13301-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 1
   [junit4]   2> 3073362 DEBUG (simCloudManagerPool-13301-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 3073362 DEBUG (simCloudManagerPool-13301-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 3073362 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 3073363 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3073363 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 3073373 DEBUG (ScheduledTrigger-13302-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073394 DEBUG (ScheduledTrigger-13302-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073415 DEBUG (ScheduledTrigger-13302-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073435 DEBUG (ScheduledTrigger-13302-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073456 DEBUG (ScheduledTrigger-13302-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073477 DEBUG (ScheduledTrigger-13302-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073497 DEBUG (ScheduledTrigger-13302-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073519 DEBUG (ScheduledTrigger-13302-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073540 DEBUG (ScheduledTrigger-13302-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073561 DEBUG (ScheduledTrigger-13302-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073563 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection .system, currentVersion=2
   [junit4]   2> 3073563 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=2
   [junit4]   2> 3073563 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 2
   [junit4]   2> 3073573 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=3
   [junit4]   2> 3073573 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 3
   [junit4]   2> 3073582 DEBUG (ScheduledTrigger-13302-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073584 DEBUG (simCloudManagerPool-13301-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=4
   [junit4]   2> 3073584 DEBUG (simCloudManagerPool-13301-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 4
   [junit4]   2> 3073585 DEBUG (simCloudManagerPool-13301-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1 (currentVersion=5): {"core_node1":{
   [junit4]   2>     "core":".system_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":".system",
   [junit4]   2>     "node_name":"127.0.0.1:10020_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 3073595 DEBUG (simCloudManagerPool-13301-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=5
   [junit4]   2> 3073595 DEBUG (simCloudManagerPool-13301-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 5
   [junit4]   2> 3073595 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection .system, currentVersion=6
   [junit4]   2> 3073597 DEBUG (simCloudManagerPool-13301-thread-5) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 3073598 DEBUG (simCloudManagerPool-13301-thread-5) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 3073598 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 3073598 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 3073598 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_trigger - Initial livenodes: [127.0.0.1:10021_solr, 127.0.0.1:10020_solr]
   [junit4]   2> 3073599 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3073599 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 3073599 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10021_solr
   [junit4]   2> 3073599 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=6
   [junit4]   2> 3073599 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 6
   [junit4]   2> 3073600 DEBUG (ScheduledTrigger-13302-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
   [junit4]   2> 3073600 DEBUG (ScheduledTrigger-13302-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10020_solr
   [junit4]   2> 3073602 INFO  (simCloudManagerPool-13301-thread-6) [    ] o.a.s.c.ActionThrottle Throttling leader attempts - waiting for 4157ms
   [junit4]   2> 3073603 DEBUG (ScheduledTrigger-13302-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 3073603 DEBUG (ScheduledTrigger-13302-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10020_solr
   [junit4]   2> 3073623 DEBUG (ScheduledTrigger-13302-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
   [junit4]   2> 3073623 DEBUG (ScheduledTrigger-13302-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 3073645 DEBUG (ScheduledTrigger-13302-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
   [junit4]   2> 3073645 DEBUG (ScheduledTrigger-13302-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 3073645 DEBUG (ScheduledTrigger-13302-thread-3) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10020_solr]
   [junit4]   2> 3073645 DEBUG (ScheduledTrigger-13302-thread-3) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"c0780a1a81a8T5f4tp5kqidui4gaktpb40lipn",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":211621798117800,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[211621798117800],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "nodeNames":["127.0.0.1:10020_solr"]}}
   [junit4]   2> 3073645 DEBUG (ScheduledTrigger-13302-thread-3) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_lost_trigger]
   [junit4]   2> 3073645 DEBUG (simCloudManagerPool-13301-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no leader in shard1:{
   [junit4]   2>   "replicas":{
   [junit4]   2>     "core_node1":{
   [junit4]   2>       "core":".system_shard1_replica_n1",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":10240,
   [junit4]   2>       "node_name":"127.0.0.1:10020_solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>       "SEARCHER.searcher.numDocs":0},
   [junit4]   2>     "core_node2":{
   [junit4]   2>       "core":".system_shard1_replica_n2",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":10240,
   [junit4]   2>       "node_name":"127.0.0.1:10021_solr",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>       "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active"}
   [junit4]   2> 3073646 DEBUG (AutoscalingActionExecutor-13303-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"c0780a1a81a8T5f4tp5kqidui4gaktpb40lipn",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":211621798117800,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[211621798117800],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":211624076802800,
   [junit4]   2>     "nodeNames":["127.0.0.1:10020_solr"]}}
   [junit4]   2> 3073647 DEBUG (simCloudManagerPool-13301-thread-8) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no leader in shard1:{
   [junit4]   2>   "replicas":{
   [junit4]   2>     "core_node1":{
   [junit4]   2>       "core":".system_shard1_replica_n1",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":10240,
   [junit4]   2>       "node_name":"127.0.0.1:10020_solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>       "SEARCHER.searcher.numDocs":0},
   [junit4]   2>     "core_node2":{
   [junit4]   2>       "core":".system_shard1_replica_n2",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":10240,
   [junit4]   2>       "node_name":"127.0.0.1:10021_solr",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>       "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active"}
   [junit4]   2> 3073647 DEBUG (AutoscalingActionExecutor-13303-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 3073647 DEBUG (AutoscalingActionExecutor-13303-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger after 100ms
   [junit4]   2> 3073647 DEBUG (AutoscalingActionExecutor-13303-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 69 ms for event id=c0780a1a81a8T5f4tp5kqidui4gaktpb40lipn
   [junit4]   2> 3073647 DEBUG (simCloudManagerPool-13301-thread-9) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 3073647 DEBUG (simCloudManagerPool-13301-thread-9) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 3073649 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 3073649 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3073649 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 3073685 DEBUG (simCloudManagerPool-13301-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1 (currentVersion=7): {"core_node2":{
   [junit4]   2>     "core":".system_shard1_replica_n2",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":".system",
   [junit4]   2>     "node_name":"127.0.0.1:10021_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 3073690 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=7
   [junit4]   2> 3073690 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 7
   [junit4]   2> 3073690 INFO  (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes: 1
   [junit4]   2> ## Empty nodes: 0
   [junit4]   2> ## Dead nodes: 1
   [junit4]   2> ## 127.0.0.1:10020_solr
   [junit4]   2> ## Collections:
   [junit4]   2> ##  * .system
   [junit4]   2> ##    shardsTotal 1
   [junit4]   2> ##    shardsState {active=1}
   [junit4]   2> ##      shardsWithoutLeader 0
   [junit4]   2> ##    totalReplicas 2
   [junit4]   2> ##      activeReplicas 2
   [junit4]   2> ##      inactiveReplicas 0
   [junit4]   2> ##    totalActiveDocs 0
   [junit4]   2> ##      bufferedDocs 0
   [junit4]   2> ##      maxActiveSliceDocs 0
   [junit4]   2> ##      minActiveSliceDocs 0
   [junit4]   2> ##      avgActiveSliceDocs 0
   [junit4]   2> ##    totalInactiveDocs 0
   [junit4]   2> ##      maxInactiveSliceDocs 0
   [junit4]   2> ##      minInactiveSliceDocs 0
   [junit4]   2> ##      avgInactiveSliceDocs NaN
   [junit4]   2> ##    totalActiveBytes 10,240
   [junit4]   2> ##      maxActiveSliceBytes 10,240
   [junit4]   2> ##      minActiveSliceBytes 10,240
   [junit4]   2> ##      avgActiveSliceBytes 10,240
   [junit4]   2> ##    totalInactiveBytes 0
   [junit4]   2> ##      maxInactiveSliceBytes 0
   [junit4]   2> ##      minInactiveSliceBytes 0
   [junit4]   2> ##      avgInactiveSliceBytes NaN
   [junit4]   2> ##    totalActiveDeletedDocs 0
   [junit4]   2> ## Max replicas per node: 1
   [junit4]   2> ## Min replicas per node: 1
   [junit4]   2> ## Total replicas: 1
   [junit4]   2> ## * .system 1
   [junit4]   2> ## - active           1
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ## - autoscaling        3
   [junit4]   2> ## - update             2
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: node_lost_trigger
   [junit4]   2> ## - STARTED         1
   [junit4]   2> ## - SUCCEEDED       1
   [junit4]   2>
   [junit4]   2> 3073694 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[1180252F246882B2]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 3073694 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10020_solr
   [junit4]   2> 3073694 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 3073694 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 3073694 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3073694 INFO  (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 3073694 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 3073695 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2070) ~[?:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189) [java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:844) [?:?]
   [junit4]   2> 3073695 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[1180252F246882B2]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 3073695 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[1180252F246882B2]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 3073695 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[1180252F246882B2]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 3073695 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[1180252F246882B2]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 3073695 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[1180252F246882B2]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 3073695 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[1180252F246882B2]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 3073735 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 3073735 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 3073736 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion -1
   [junit4]   2> 3073736 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 3073736 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10022_solr, 127.0.0.1:10021_solr]
   [junit4]   2> 3073736 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3073736 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 3073736 DEBUG (ScheduledTrigger-13305-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073758 DEBUG (ScheduledTrigger-13305-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073779 DEBUG (ScheduledTrigger-13305-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073800 DEBUG (ScheduledTrigger-13305-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073822 DEBUG (ScheduledTrigger-13305-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073843 DEBUG (ScheduledTrigger-13305-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073865 DEBUG (ScheduledTrigger-13305-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073886 DEBUG (ScheduledTrigger-13305-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073896 DEBUG (simCloudManagerPool-13304-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=8
   [junit4]   2> 3073896 DEBUG (simCloudManagerPool-13304-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 8
   [junit4]   2> 3073896 DEBUG (simCloudManagerPool-13304-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 3073896 DEBUG (simCloudManagerPool-13304-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
   [junit4]   2> 3073897 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 3073899 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3073899 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 3073907 DEBUG (ScheduledTrigger-13305-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073928 DEBUG (ScheduledTrigger-13305-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073950 DEBUG (ScheduledTrigger-13305-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073971 DEBUG (ScheduledTrigger-13305-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3073991 DEBUG (ScheduledTrigger-13305-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3074012 DEBUG (ScheduledTrigger-13305-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3074033 DEBUG (ScheduledTrigger-13305-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3074054 DEBUG (ScheduledTrigger-13305-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3074075 DEBUG (ScheduledTrigger-13305-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3074096 DEBUG (ScheduledTrigger-13305-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3074098 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection .system, currentVersion=9
   [junit4]   2> 3074098 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=9
   [junit4]   2> 3074098 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 9
   [junit4]   2> 3074108 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=10
   [junit4]   2> 3074108 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 10
   [junit4]   2> 3074117 DEBUG (ScheduledTrigger-13305-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3074120 DEBUG (simCloudManagerPool-13304-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=11
   [junit4]   2> 3074120 DEBUG (simCloudManagerPool-13304-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 11
   [junit4]   2> 3074121 DEBUG (simCloudManagerPool-13304-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1 (currentVersion=12): {"core_node3":{
   [junit4]   2>     "core":".system_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":".system",
   [junit4]   2>     "node_name":"127.0.0.1:10021_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 3074131 DEBUG (simCloudManagerPool-13304-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=12
   [junit4]   2> 3074131 DEBUG (simCloudManagerPool-13304-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 12
   [junit4]   2> 3074131 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[1180252F246882B2]) [    ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection .system, currentVersion=13
   [junit4]   2> 3074133 DEBUG (simCloudManagerPool-13304-thread-5) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 3074133 DEBUG (simCloudManagerPool-13304-thread-5) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
   [junit4]   2> 3074133 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 3074133 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 3074133 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - Initial livenodes: [127.0.0.1:10022_solr, 127.0.0.1:10021_solr]
   [junit4]   2> 3074134 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=10, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestTriggerAction}], enabled=true}
   [junit4]   2> 3074134 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 3074134 DEBUG (ScheduledTrigger-13305-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3074134 DEBUG (ScheduledTrigger-13305-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3074134 DEBUG (ScheduledTrigger-13305-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10023_solr at time 211648502907800
   [junit4]   2> 3074138 DEBUG (ScheduledTrigger-13305-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [juni

[...truncated too long message...]

 codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=354, maxMBSortInHeap=5.001178232677161, sim=RandomSimilarity(queryNorm=false): {}, locale=as-IN, timezone=America/Blanc-Sablon
   [junit4]   2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 10.0.1 (64-bit)/cpus=3,threads=3,free=123925560,total=231735296
   [junit4]   2> NOTE: All tests run in this JVM: [TestManagedResourceStorage]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestManagedResourceStorage -Dtests.seed=1180252F246882B2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=as-IN -Dtests.timezone=America/Blanc-Sablon -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J1 | TestManagedResourceStorage (suite) <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: SolrCore.getOpenCount()==2
   [junit4]    > at __randomizedtesting.SeedInfo.seed([1180252F246882B2]:0)
   [junit4]    > at org.apache.solr.util.TestHarness.close(TestHarness.java:380)
   [junit4]    > at org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:802)
   [junit4]    > at org.apache.solr.cloud.AbstractZkTestCase.azt_afterClass(AbstractZkTestCase.java:147)
   [junit4]    > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   [junit4]    > at java.base/java.lang.Thread.run(Thread.java:844)Throwable #2: java.io.IOException: Could not remove the following files (in the order of attempts):
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2\log.1: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2\log.1: The process cannot access the file because it is being used by another process.
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001\version-2
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001\zkData-001
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.rest.TestManagedResourceStorage_1180252F246882B2-001
   [junit4]    > at org.apache.lucene.util.IOUtils.rm(IOUtils.java:318)
   [junit4]    > at java.base/java.lang.Thread.run(Thread.java:844)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.rest.TestManagedResourceStorage:
   [junit4]    >    1) Thread[id=50, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestManagedResourceStorage]
   [junit4]    >         at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
   [junit4]    >         at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
   [junit4]    >         at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
   [junit4]    >         at java.base@10.0.1/sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
   [junit4]    >         at java.base@10.0.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:89)
   [junit4]    >         at java.base@10.0.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:100)
   [junit4]    >         at app//org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196)
   [junit4]    >         at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
   [junit4]    >    2) Thread[id=51, name=SessionTracker, state=TIMED_WAITING, group=TGRP-TestManagedResourceStorage]
   [junit4]    >         at java.base@10.0.1/java.lang.Object.wait(Native Method)
   [junit4]    >         at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:147)
   [junit4]    >    3) Thread[id=49, name=Thread-0, state=WAITING, group=TGRP-TestManagedResourceStorage]
   [junit4]    >         at java.base@10.0.1/java.lang.Object.wait(Native Method)
   [junit4]    >         at java.base@10.0.1/java.lang.Thread.join(Thread.java:1353)
   [junit4]    >         at java.base@10.0.1/java.lang.Thread.join(Thread.java:1427)
   [junit4]    >         at app//org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:313)
   [junit4]    >         at app//org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:313)
   [junit4]    >         at app//org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:496)
   [junit4]    >    4) Thread[id=53, name=ProcessThread(sid:0 cport:53355):, state=WAITING, group=TGRP-TestManagedResourceStorage]
   [junit4]    >         at java.base@10.0.1/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@10.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
   [junit4]    >         at java.base@10.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2075)
   [junit4]    >         at java.base@10.0.1/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
   [junit4]    >         at app//org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:123)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([1180252F246882B2]:0)Throwable #4: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=50, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestManagedResourceStorage]
   [junit4]    >         at java.base@10.0.1/sun.nio.ch.Util$2.toArray(Util.java:342)
   [junit4]    >         at java.base@10.0.1/java.util.ArrayList.<init>(ArrayList.java:179)
   [junit4]    >         at app//org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:201)
   [junit4]    >         at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([1180252F246882B2]:0)
   [junit4] Completed [12/15 (1!)] on J1 in 97.39s, 2 tests, 4 errors <<< FAILURES!

[...truncated 19 lines...]
BUILD FAILED
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\common-build.xml:1567: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\common-build.xml:1092: There were test failures: 15 suites, 90 tests, 4 suite-level errors, 24 ignored [seed: 1180252F246882B2]

Total time: 1 minute 41 seconds

[repro] Setting last failure code to 1

[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue
[repro]   0/5 failed: org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
[repro]   1/5 failed: org.apache.solr.rest.TestManagedResourceStorage
[repro] Exiting with code 1

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>IF EXIST lucene\build move /y lucene\build lucene\build.repro
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>IF EXIST solr\build move /y solr\build solr\build.repro
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>IF EXIST lucene\build.orig move /y lucene\build.orig lucene\build
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>IF EXIST solr\build.orig move /y solr\build.orig solr\build
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>exit 0
Archiving artifacts
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=76271d5a7f40ec9b5a1ac9e723fa92f809ad8f12, workspace=C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows
[WARNINGS] Computing warning deltas based on reference build #828
Recording test results
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS-EA] Lucene-Solr-7.x-Windows (64bit/jdk-12-ea+12) - Build # 830 - Failure!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Windows/830/
Java: 64bit/jdk-12-ea+12 -XX:+UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.DeleteShardTest.testDirectoryCleanupAfterDeleteShard

Error Message:
Could not find collection : deleteshard_test

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : deleteshard_test
        at __randomizedtesting.SeedInfo.seed([8C6EDDB315F284EC:2C7496E8B182CFE0]:0)
        at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:118)
        at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:258)
        at org.apache.solr.cloud.DeleteShardTest.testDirectoryCleanupAfterDeleteShard(DeleteShardTest.java:114)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:835)




Build Log:
[...truncated 13892 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteShardTest
   [junit4]   2> 2131464 INFO  (SUITE-DeleteShardTest-seed#[8C6EDDB315F284EC]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.DeleteShardTest_8C6EDDB315F284EC-001\init-core-data-001
   [junit4]   2> 2131465 WARN  (SUITE-DeleteShardTest-seed#[8C6EDDB315F284EC]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 2131465 INFO  (SUITE-DeleteShardTest-seed#[8C6EDDB315F284EC]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2131465 INFO  (SUITE-DeleteShardTest-seed#[8C6EDDB315F284EC]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0)
   [junit4]   2> 2131470 INFO  (SUITE-DeleteShardTest-seed#[8C6EDDB315F284EC]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.DeleteShardTest_8C6EDDB315F284EC-001\tempDir-001
   [junit4]   2> 2131470 INFO  (SUITE-DeleteShardTest-seed#[8C6EDDB315F284EC]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2131470 INFO  (Thread-2568) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2131470 INFO  (Thread-2568) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2131476 ERROR (Thread-2568) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2131641 INFO  (SUITE-DeleteShardTest-seed#[8C6EDDB315F284EC]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:64180
   [junit4]   2> 2131662 INFO  (zkConnectionManagerCallback-7809-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2131741 INFO  (jetty-launcher-7806-thread-1) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 12-ea+12
   [junit4]   2> 2131741 INFO  (jetty-launcher-7806-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2131741 INFO  (jetty-launcher-7806-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2131741 INFO  (jetty-launcher-7806-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2131742 INFO  (jetty-launcher-7806-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6b8f1854{/solr,null,AVAILABLE}
   [junit4]   2> 2131743 INFO  (jetty-launcher-7806-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7431dd78{SSL,[ssl, http/1.1]}{127.0.0.1:64184}
   [junit4]   2> 2131743 INFO  (jetty-launcher-7806-thread-1) [    ] o.e.j.s.Server Started @2131777ms
   [junit4]   2> 2131743 INFO  (jetty-launcher-7806-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64184}
   [junit4]   2> 2131743 ERROR (jetty-launcher-7806-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2131743 INFO  (jetty-launcher-7806-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2131743 INFO  (jetty-launcher-7806-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 2131743 INFO  (jetty-launcher-7806-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2131743 INFO  (jetty-launcher-7806-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2131743 INFO  (jetty-launcher-7806-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-11T02:39:36.199554700Z
   [junit4]   2> 2131747 INFO  (jetty-launcher-7806-thread-2) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 12-ea+12
   [junit4]   2> 2131747 INFO  (jetty-launcher-7806-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2131747 INFO  (jetty-launcher-7806-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2131747 INFO  (jetty-launcher-7806-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2131766 INFO  (jetty-launcher-7806-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@52cf5c5e{/solr,null,AVAILABLE}
   [junit4]   2> 2131767 INFO  (jetty-launcher-7806-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@b34e53b{SSL,[ssl, http/1.1]}{127.0.0.1:64189}
   [junit4]   2> 2131767 INFO  (jetty-launcher-7806-thread-2) [    ] o.e.j.s.Server Started @2131801ms
   [junit4]   2> 2131767 INFO  (jetty-launcher-7806-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64189}
   [junit4]   2> 2131767 ERROR (jetty-launcher-7806-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2131767 INFO  (jetty-launcher-7806-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2131767 INFO  (jetty-launcher-7806-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 2131767 INFO  (jetty-launcher-7806-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2131767 INFO  (jetty-launcher-7806-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2131767 INFO  (jetty-launcher-7806-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-11T02:39:36.223705600Z
   [junit4]   2> 2131786 INFO  (zkConnectionManagerCallback-7811-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2131787 INFO  (zkConnectionManagerCallback-7813-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2131787 INFO  (jetty-launcher-7806-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2131787 INFO  (jetty-launcher-7806-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2131797 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1000685d53e0001, likely client has closed socket
   [junit4]   2> 2132430 INFO  (jetty-launcher-7806-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64180/solr
   [junit4]   2> 2132453 INFO  (zkConnectionManagerCallback-7817-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2132455 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1000685d53e0003, likely client has closed socket
   [junit4]   2> 2132458 INFO  (zkConnectionManagerCallback-7819-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2132696 INFO  (jetty-launcher-7806-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64180/solr
   [junit4]   2> 2132704 INFO  (zkConnectionManagerCallback-7825-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2132713 INFO  (zkConnectionManagerCallback-7827-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2132728 INFO  (jetty-launcher-7806-thread-2) [n:127.0.0.1:64189_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:64189_solr
   [junit4]   2> 2132730 INFO  (jetty-launcher-7806-thread-2) [n:127.0.0.1:64189_solr    ] o.a.s.c.Overseer Overseer (id=72064765915955206-127.0.0.1:64189_solr-n_0000000000) starting
   [junit4]   2> 2132736 INFO  (jetty-launcher-7806-thread-1) [n:127.0.0.1:64184_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64184_solr
   [junit4]   2> 2132771 INFO  (zkCallback-7826-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2132771 INFO  (zkCallback-7818-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2132772 INFO  (zkConnectionManagerCallback-7839-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2132773 INFO  (zkConnectionManagerCallback-7834-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2132773 INFO  (jetty-launcher-7806-thread-1) [n:127.0.0.1:64184_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2132774 INFO  (jetty-launcher-7806-thread-2) [n:127.0.0.1:64189_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2132774 INFO  (jetty-launcher-7806-thread-1) [n:127.0.0.1:64184_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:64180/solr ready
   [junit4]   2> 2132775 INFO  (jetty-launcher-7806-thread-1) [n:127.0.0.1:64184_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2132775 INFO  (jetty-launcher-7806-thread-2) [n:127.0.0.1:64189_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:64180/solr ready
   [junit4]   2> 2132777 INFO  (jetty-launcher-7806-thread-2) [n:127.0.0.1:64189_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2132777 INFO  (jetty-launcher-7806-thread-2) [n:127.0.0.1:64189_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64189_solr
   [junit4]   2> 2132778 INFO  (zkCallback-7826-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2132778 INFO  (zkCallback-7818-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2132804 INFO  (OverseerStateUpdate-72064765915955206-127.0.0.1:64189_solr-n_0000000000) [n:127.0.0.1:64189_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:64189_solr
   [junit4]   2> 2132809 INFO  (jetty-launcher-7806-thread-1) [n:127.0.0.1:64184_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64184.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68ba1510
   [junit4]   2> 2132809 INFO  (jetty-launcher-7806-thread-2) [n:127.0.0.1:64189_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2132819 INFO  (jetty-launcher-7806-thread-1) [n:127.0.0.1:64184_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64184.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68ba1510
   [junit4]   2> 2132819 INFO  (jetty-launcher-7806-thread-1) [n:127.0.0.1:64184_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64184.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68ba1510
   [junit4]   2> 2132821 INFO  (jetty-launcher-7806-thread-1) [n:127.0.0.1:64184_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.DeleteShardTest_8C6EDDB315F284EC-001\tempDir-001\node1\.
   [junit4]   2> 2132824 INFO  (zkCallback-7838-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2132838 INFO  (zkCallback-7833-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2132838 INFO  (jetty-launcher-7806-thread-2) [n:127.0.0.1:64189_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64189.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68ba1510
   [junit4]   2> 2132863 INFO  (jetty-launcher-7806-thread-2) [n:127.0.0.1:64189_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64189.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68ba1510
   [junit4]   2> 2132864 INFO  (jetty-launcher-7806-thread-2) [n:127.0.0.1:64189_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64189.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68ba1510
   [junit4]   2> 2132865 INFO  (jetty-launcher-7806-thread-2) [n:127.0.0.1:64189_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.DeleteShardTest_8C6EDDB315F284EC-001\tempDir-001\node2\.
   [junit4]   2> 2132913 INFO  (zkConnectionManagerCallback-7843-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2132922 INFO  (zkConnectionManagerCallback-7848-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2132923 INFO  (SUITE-DeleteShardTest-seed#[8C6EDDB315F284EC]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2132925 INFO  (SUITE-DeleteShardTest-seed#[8C6EDDB315F284EC]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:64180/solr ready
   [junit4]   2> 2132956 INFO  (TEST-DeleteShardTest.testDirectoryCleanupAfterDeleteShard-seed#[8C6EDDB315F284EC]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDirectoryCleanupAfterDeleteShard
   [junit4]   2> 2132991 INFO  (qtp1868790745-17498) [n:127.0.0.1:64184_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params shards=a,b,c&collection.configName=conf&maxShardsPerNode=2&name=deleteshard_test&router.name=implicit&nrtReplicas=1&action=CREATE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2133003 INFO  (OverseerThreadFactory-5998-thread-1-processing-n:127.0.0.1:64189_solr) [n:127.0.0.1:64189_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection deleteshard_test
   [junit4]   2> 2133178 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=4
   [junit4]   2> 2133226 INFO  (qtp96379716-17516) [n:127.0.0.1:64189_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=3
   [junit4]   2> 2133302 INFO  (OverseerStateUpdate-72064765915955206-127.0.0.1:64189_solr-n_0000000000) [n:127.0.0.1:64189_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deleteshard_test",
   [junit4]   2>   "shard":"a",
   [junit4]   2>   "core":"deleteshard_test_a_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:64189/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2133311 INFO  (OverseerStateUpdate-72064765915955206-127.0.0.1:64189_solr-n_0000000000) [n:127.0.0.1:64189_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deleteshard_test",
   [junit4]   2>   "shard":"b",
   [junit4]   2>   "core":"deleteshard_test_b_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:64184/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2133337 INFO  (OverseerStateUpdate-72064765915955206-127.0.0.1:64189_solr-n_0000000000) [n:127.0.0.1:64189_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deleteshard_test",
   [junit4]   2>   "shard":"c",
   [junit4]   2>   "core":"deleteshard_test_c_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:64184/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2133551 WARN  (qtp1868790745-17501) [n:127.0.0.1:64184_solr    ] o.e.j.u.t.s.EatWhatYouKill
   [junit4]   2> java.util.NoSuchElementException: No value present
   [junit4]   2> at java.util.Optional.get(Optional.java:148) ~[?:?]
   [junit4]   2> at sun.security.ssl.ServerHello$T13ServerHelloProducer.produce(ServerHello.java:551) ~[?:?]
   [junit4]   2> at sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436) ~[?:?]
   [junit4]   2> at sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(ClientHello.java:1189) ~[?:?]
   [junit4]   2> at sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1125) ~[?:?]
   [junit4]   2> at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:831) ~[?:?]
   [junit4]   2> at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:792) ~[?:?]
   [junit4]   2> at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
   [junit4]   2> at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:?]
   [junit4]   2> at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1065) ~[?:?]
   [junit4]   2> at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1052) ~[?:?]
   [junit4]   2> at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
   [junit4]   2> at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:999) ~[?:?]
   [junit4]   2> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:775) ~[jetty-io-9.4.11.v20180605.jar:9.4.11.v20180605]
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:331) ~[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:243) ~[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) ~[jetty-io-9.4.11.v20180605.jar:9.4.11.v20180605]
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) ~[jetty-io-9.4.11.v20180605.jar:9.4.11.v20180605]
   [junit4]   2> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291) ~[jetty-io-9.4.11.v20180605.jar:9.4.11.v20180605]
   [junit4]   2> at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151) ~[jetty-io-9.4.11.v20180605.jar:9.4.11.v20180605]
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) ~[jetty-io-9.4.11.v20180605.jar:9.4.11.v20180605]
   [junit4]   2> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) ~[jetty-io-9.4.11.v20180605.jar:9.4.11.v20180605]
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605]
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605]
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605]
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605]
   [junit4]   2> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605]
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605]
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605]
   [junit4]   2> at java.lang.Thread.run(Thread.java:835) [?:?]
   [junit4]   2> 2133552 ERROR (OverseerThreadFactory-5998-thread-1-processing-n:127.0.0.1:64189_solr) [n:127.0.0.1:64189_solr    ] o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard: https://127.0.0.1:64184/solr
   [junit4]   2> org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:64184/solr
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219) ~[java/:?]
   [junit4]   2> at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:172) ~[java/:?]
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
   [junit4]   2> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) ~[metrics-core-3.2.6.jar:3.2.6]
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) [java/:?]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:835) [?:?]
   [junit4]   2> Caused by: java.net.SocketException: Socket input is already shutdown
   [junit4]   2> at sun.security.ssl.SSLSocketImpl.getInputStream(SSLSocketImpl.java:696) ~[?:?]
   [junit4]   2> at org.apache.http.impl.BHttpConnectionBase.getSocketInputStream(BHttpConnectionBase.java:142) ~[httpcore-4.4.6.jar:4.4.6]
   [junit4]   2> at org.apache.http.impl.conn.LoggingManagedHttpClientConnection.getSocketInputStream(LoggingManagedHttpClientConnection.java:103) ~[httpclient-4.5.3.jar:4.5.3]
   [junit4]   2> at org.apache.http.impl.BHttpConnectionBase.ensureOpen(BHttpConnectionBase.java:134) ~[httpcore-4.4.6.jar:4.4.6]
   [junit4]   2> at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestHeader(DefaultBHttpClientConnection.java:140) ~[httpcore-4.4.6.jar:4.4.6]
   [junit4]   2> at org.apache.http.impl.conn.CPoolProxy.sendRequestHeader(CPoolProxy.java:155) ~[httpclient-4.5.3.jar:4.5.3]
   [junit4]   2> at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:205) ~[httpcore-4.4.6.jar:4.4.6]
   [junit4]   2> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) ~[httpcore-4.4.6.jar:4.4.6]
   [junit4]   2> at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:120) ~[java/:?]
   [junit4]   2> at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.3.jar:4.5.3]
   [junit4]   2> at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.3.jar:4.5.3]
   [junit4]   2> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.3.jar:4.5.3]
   [junit4]   2> at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[httpclient-4.5.3.jar:4.5.3]
   [junit4]   2> at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.3.jar:4.5.3]
   [junit4]   2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.3.jar:4.5.3]
   [junit4]   2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.3.jar:4.5.3]
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542) ~[java/:?]
   [junit4]   2> ... 12 more
   [junit4]   2> 2133559 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr    x:deleteshard_test_b_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=deleteshard_test_b_replica_n2&action=CREATE&numShards=3&collection=deleteshard_test&shard=b&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2133560 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr    x:deleteshard_test_b_replica_n2] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2133568 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr    x:deleteshard_test_a_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=deleteshard_test_a_replica_n1&action=CREATE&numShards=3&collection=deleteshard_test&shard=a&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2134589 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 2134599 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.s.IndexSchema [deleteshard_test_b_replica_n2] Schema name=minimal
   [junit4]   2> 2134601 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2134601 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'deleteshard_test_b_replica_n2' using configuration from collection deleteshard_test, trusted=true
   [junit4]   2> 2134602 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64184.solr.core.deleteshard_test.b.replica_n2' (registry 'solr.core.deleteshard_test.b.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68ba1510
   [junit4]   2> 2134602 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.c.SolrCore [[deleteshard_test_b_replica_n2] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.DeleteShardTest_8C6EDDB315F284EC-001\tempDir-001\node1\deleteshard_test_b_replica_n2], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.DeleteShardTest_8C6EDDB315F284EC-001\tempDir-001\node1\.\deleteshard_test_b_replica_n2\data\]
   [junit4]   2> 2134613 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 2134621 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.s.IndexSchema [deleteshard_test_a_replica_n1] Schema name=minimal
   [junit4]   2> 2134633 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2134633 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'deleteshard_test_a_replica_n1' using configuration from collection deleteshard_test, trusted=true
   [junit4]   2> 2134634 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64189.solr.core.deleteshard_test.a.replica_n1' (registry 'solr.core.deleteshard_test.a.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68ba1510
   [junit4]   2> 2134635 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.c.SolrCore [[deleteshard_test_a_replica_n1] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.DeleteShardTest_8C6EDDB315F284EC-001\tempDir-001\node2\deleteshard_test_a_replica_n1], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.DeleteShardTest_8C6EDDB315F284EC-001\tempDir-001\node2\.\deleteshard_test_a_replica_n1\data\]
   [junit4]   2> 2134741 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2134741 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2134744 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2134744 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2134757 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@59137e[deleteshard_test_b_replica_n2] main]
   [junit4]   2> 2134758 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2134759 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2134759 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2134760 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613995000951996416
   [junit4]   2> 2134760 INFO  (searcherExecutor-6003-thread-1-processing-n:127.0.0.1:64184_solr x:deleteshard_test_b_replica_n2 c:deleteshard_test s:b r:core_node5) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.c.SolrCore [deleteshard_test_b_replica_n2] Registered new searcher Searcher@59137e[deleteshard_test_b_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2134767 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/deleteshard_test/terms/b to Terms{values={core_node5=0}, version=0}
   [junit4]   2> 2134771 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2134771 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2134771 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:64184/solr/deleteshard_test_b_replica_n2/
   [junit4]   2> 2134771 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2134772 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.c.SyncStrategy https://127.0.0.1:64184/solr/deleteshard_test_b_replica_n2/ has no replicas
   [junit4]   2> 2134772 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2134775 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:64184/solr/deleteshard_test_b_replica_n2/ b
   [junit4]   2> 2134781 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2134781 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2134785 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2134785 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2134786 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7ece6ef6[deleteshard_test_a_replica_n1] main]
   [junit4]   2> 2134788 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2134788 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2134789 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2134789 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613995000982405120
   [junit4]   2> 2134789 INFO  (searcherExecutor-6004-thread-1-processing-n:127.0.0.1:64189_solr x:deleteshard_test_a_replica_n1 c:deleteshard_test s:a r:core_node3) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.c.SolrCore [deleteshard_test_a_replica_n1] Registered new searcher Searcher@7ece6ef6[deleteshard_test_a_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2134794 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/deleteshard_test/terms/a to Terms{values={core_node3=0}, version=0}
   [junit4]   2> 2134798 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2134798 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2134798 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:64189/solr/deleteshard_test_a_replica_n1/
   [junit4]   2> 2134799 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2134799 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:64189/solr/deleteshard_test_a_replica_n1/ has no replicas
   [junit4]   2> 2134799 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2134802 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:64189/solr/deleteshard_test_a_replica_n1/ a
   [junit4]   2> 2134934 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2134939 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteshard_test s:b r:core_node5 x:deleteshard_test_b_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=deleteshard_test_b_replica_n2&action=CREATE&numShards=3&collection=deleteshard_test&shard=b&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1379
   [junit4]   2> 2134957 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2134960 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr c:deleteshard_test s:a r:core_node3 x:deleteshard_test_a_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=deleteshard_test_a_replica_n1&action=CREATE&numShards=3&collection=deleteshard_test&shard=a&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1393
   [junit4]   2> 2134962 ERROR (OverseerThreadFactory-5998-thread-1-processing-n:127.0.0.1:64189_solr) [n:127.0.0.1:64189_solr    ] o.a.s.c.a.c.OverseerCollectionMessageHandler Cleaning up collection [deleteshard_test].
   [junit4]   2> 2134962 INFO  (OverseerThreadFactory-5998-thread-1-processing-n:127.0.0.1:64189_solr) [n:127.0.0.1:64189_solr    ] o.a.s.c.a.c.OverseerCollectionMessageHandler Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true&deleteMetricsHistory=true, asyncId=null
   [junit4]   2> 2134964 INFO  (qtp1868790745-17504) [n:127.0.0.1:64184_solr    x:deleteshard_test_b_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.deleteshard_test.b.replica_n2, tag=null
   [junit4]   2> 2134964 INFO  (qtp1868790745-17504) [n:127.0.0.1:64184_solr    x:deleteshard_test_b_replica_n2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@652c5cd: rootName = solr_64184, domain = solr.core.deleteshard_test.b.replica_n2, service url = null, agent id = null] for registry solr.core.deleteshard_test.b.replica_n2 / com.codahale.metrics.MetricRegistry@c88f74b
   [junit4]   2> 2134964 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr    x:deleteshard_test_a_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.deleteshard_test.a.replica_n1, tag=null
   [junit4]   2> 2134964 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr    x:deleteshard_test_a_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5b6e46be: rootName = solr_64189, domain = solr.core.deleteshard_test.a.replica_n1, service url = null, agent id = null] for registry solr.core.deleteshard_test.a.replica_n1 / com.codahale.metrics.MetricRegistry@19a5c9b9
   [junit4]   2> 2134979 INFO  (qtp1868790745-17504) [n:127.0.0.1:64184_solr    x:deleteshard_test_b_replica_n2] o.a.s.c.SolrCore [deleteshard_test_b_replica_n2]  CLOSING SolrCore org.apache.solr.core.SolrCore@71b27da7
   [junit4]   2> 2134979 INFO  (qtp1868790745-17504) [n:127.0.0.1:64184_solr    x:deleteshard_test_b_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.deleteshard_test.b.replica_n2, tag=71b27da7
   [junit4]   2> 2134979 INFO  (qtp1868790745-17504) [n:127.0.0.1:64184_solr    x:deleteshard_test_b_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.deleteshard_test.b.leader, tag=71b27da7
   [junit4]   2> 2134993 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr    x:deleteshard_test_a_replica_n1] o.a.s.c.SolrCore [deleteshard_test_a_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1fb77cbc
   [junit4]   2> 2134993 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr    x:deleteshard_test_a_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.deleteshard_test.a.replica_n1, tag=1fb77cbc
   [junit4]   2> 2134993 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr    x:deleteshard_test_a_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.deleteshard_test.a.leader, tag=1fb77cbc
   [junit4]   2> 2135005 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr    x:deleteshard_test_a_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/deleteshard_test/terms/a to Terms{values={}, version=1}
   [junit4]   2> 2135005 INFO  (qtp1868790745-17504) [n:127.0.0.1:64184_solr    x:deleteshard_test_b_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/deleteshard_test/terms/b to Terms{values={}, version=1}
   [junit4]   2> 2135009 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr    x:deleteshard_test_a_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&deleteMetricsHistory=true&core=deleteshard_test_a_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=46
   [junit4]   2> 2135010 INFO  (qtp1868790745-17504) [n:127.0.0.1:64184_solr    x:deleteshard_test_b_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&deleteMetricsHistory=true&core=deleteshard_test_b_replica_n2&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=47
   [junit4]   2> 2135114 INFO  (zkCallback-7826-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/deleteshard_test/state.json] for collection [deleteshard_test] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2135114 INFO  (zkCallback-7818-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/deleteshard_test/state.json] for collection [deleteshard_test] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2135721 INFO  (OverseerThreadFactory-5998-thread-1-processing-n:127.0.0.1:64189_solr) [n:127.0.0.1:64189_solr    ] o.a.s.c.a.c.CreateCollectionCmd Cleaned up artifacts for failed create collection for [deleteshard_test]
   [junit4]   2> 2135724 INFO  (qtp1868790745-17498) [n:127.0.0.1:64184_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2137015 INFO  (OverseerCollectionConfigSetProcessor-72064765915955206-127.0.0.1:64189_solr-n_0000000000) [n:127.0.0.1:64189_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2165760 ERROR (qtp1868790745-17498) [n:127.0.0.1:64184_solr    ] o.a.s.h.a.CollectionsHandler Timed out waiting for new collection's replicas to become ACTIVE with timeout=30
   [junit4]   2> 2165760 INFO  (qtp1868790745-17498) [n:127.0.0.1:64184_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={shards=a,b,c&collection.configName=conf&maxShardsPerNode=2&name=deleteshard_test&router.name=implicit&nrtReplicas=1&action=CREATE&wt=javabin&version=2} status=0 QTime=32768
   [junit4]   2> 2165761 INFO  (TEST-DeleteShardTest.testDirectoryCleanupAfterDeleteShard-seed#[8C6EDDB315F284EC]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDirectoryCleanupAfterDeleteShard
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteShardTest -Dtests.method=testDirectoryCleanupAfterDeleteShard -Dtests.seed=8C6EDDB315F284EC -Dtests.slow=true -Dtests.locale=gv -Dtests.timezone=Asia/Qatar -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   32.8s J1 | DeleteShardTest.testDirectoryCleanupAfterDeleteShard <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: Could not find collection : deleteshard_test
   [junit4]    > at __randomizedtesting.SeedInfo.seed([8C6EDDB315F284EC:2C7496E8B182CFE0]:0)
   [junit4]    > at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:118)
   [junit4]    > at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:258)
   [junit4]    > at org.apache.solr.cloud.DeleteShardTest.testDirectoryCleanupAfterDeleteShard(DeleteShardTest.java:114)
   [junit4]    > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   [junit4]    > at java.base/java.lang.Thread.run(Thread.java:835)
   [junit4]   2> 2165776 INFO  (TEST-DeleteShardTest.test-seed#[8C6EDDB315F284EC]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 2165776 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf&name=deleteShard&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2165781 INFO  (OverseerThreadFactory-5998-thread-2-processing-n:127.0.0.1:64189_solr) [n:127.0.0.1:64189_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection deleteShard
   [junit4]   2> 2165891 INFO  (qtp1868790745-17504) [n:127.0.0.1:64184_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=3
   [junit4]   2> 2165896 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=3
   [junit4]   2> 2165902 INFO  (OverseerStateUpdate-72064765915955206-127.0.0.1:64189_solr-n_0000000000) [n:127.0.0.1:64189_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deleteShard",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"deleteShard_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:64189/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2165904 INFO  (OverseerStateUpdate-72064765915955206-127.0.0.1:64189_solr-n_0000000000) [n:127.0.0.1:64189_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deleteShard",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"deleteShard_shard2_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:64184/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2166107 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr    x:deleteShard_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=deleteShard_shard1_replica_n1&action=CREATE&numShards=2&collection=deleteShard&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2166107 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr    x:deleteShard_shard2_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=deleteShard_shard2_replica_n2&action=CREATE&numShards=2&collection=deleteShard&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2167119 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 2167119 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 2167125 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.s.IndexSchema [deleteShard_shard2_replica_n2] Schema name=minimal
   [junit4]   2> 2167125 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.s.IndexSchema [deleteShard_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 2167127 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2167127 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'deleteShard_shard2_replica_n2' using configuration from collection deleteShard, trusted=true
   [junit4]   2> 2167127 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2167127 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'deleteShard_shard1_replica_n1' using configuration from collection deleteShard, trusted=true
   [junit4]   2> 2167127 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64189.solr.core.deleteShard.shard1.replica_n1' (registry 'solr.core.deleteShard.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68ba1510
   [junit4]   2> 2167127 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64184.solr.core.deleteShard.shard2.replica_n2' (registry 'solr.core.deleteShard.shard2.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68ba1510
   [junit4]   2> 2167127 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.c.SolrCore [[deleteShard_shard2_replica_n2] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.DeleteShardTest_8C6EDDB315F284EC-001\tempDir-001\node1\deleteShard_shard2_replica_n2], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.DeleteShardTest_8C6EDDB315F284EC-001\tempDir-001\node1\.\deleteShard_shard2_replica_n2\data\]
   [junit4]   2> 2167127 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.c.SolrCore [[deleteShard_shard1_replica_n1] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.DeleteShardTest_8C6EDDB315F284EC-001\tempDir-001\node2\deleteShard_shard1_replica_n1], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.DeleteShardTest_8C6EDDB315F284EC-001\tempDir-001\node2\.\deleteShard_shard1_replica_n1\data\]
   [junit4]   2> 2167226 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2167226 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2167229 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2167229 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2167230 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2167230 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2167232 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2167232 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2167232 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@5f73b88e[deleteShard_shard2_replica_n2] main]
   [junit4]   2> 2167234 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2167234 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2167235 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2167235 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613995035004502016
   [junit4]   2> 2167236 INFO  (searcherExecutor-6013-thread-1-processing-n:127.0.0.1:64184_solr x:deleteShard_shard2_replica_n2 c:deleteShard s:shard2 r:core_node4) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.c.SolrCore [deleteShard_shard2_replica_n2] Registered new searcher Searcher@5f73b88e[deleteShard_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2167237 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@720b09e2[deleteShard_shard1_replica_n1] main]
   [junit4]   2> 2167240 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2167240 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2167240 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2167242 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613995035009744896
   [junit4]   2> 2167242 INFO  (searcherExecutor-6014-thread-1-processing-n:127.0.0.1:64189_solr x:deleteShard_shard1_replica_n1 c:deleteShard s:shard1 r:core_node3) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.c.SolrCore [deleteShard_shard1_replica_n1] Registered new searcher Searcher@720b09e2[deleteShard_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2167245 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/deleteShard/terms/shard2 to Terms{values={core_node4=0}, version=0}
   [junit4]   2> 2167248 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/deleteShard/terms/shard1 to Terms{values={core_node3=0}, version=0}
   [junit4]   2> 2167250 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2167250 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2167250 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:64184/solr/deleteShard_shard2_replica_n2/
   [junit4]   2> 2167250 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2167250 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.c.SyncStrategy https://127.0.0.1:64184/solr/deleteShard_shard2_replica_n2/ has no replicas
   [junit4]   2> 2167251 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2167251 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2167251 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2167251 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:64189/solr/deleteShard_shard1_replica_n1/
   [junit4]   2> 2167251 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2167253 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:64189/solr/deleteShard_shard1_replica_n1/ has no replicas
   [junit4]   2> 2167253 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2167256 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:64184/solr/deleteShard_shard2_replica_n2/ shard2
   [junit4]   2> 2167257 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:64189/solr/deleteShard_shard1_replica_n1/ shard1
   [junit4]   2> 2167409 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2167414 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2167414 INFO  (qtp1868790745-17505) [n:127.0.0.1:64184_solr c:deleteShard s:shard2 r:core_node4 x:deleteShard_shard2_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=deleteShard_shard2_replica_n2&action=CREATE&numShards=2&collection=deleteShard&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1306
   [junit4]   2> 2167417 INFO  (qtp96379716-17513) [n:127.0.0.1:64189_solr c:deleteShard s:shard1 r:core_node3 x:deleteShard_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=deleteShard_shard1_replica_n1&action=CREATE&numShards=2&collection=deleteShard&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1309
   [junit4]   2> 2167419 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2167517 INFO  (zkCallback-7818-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteShard/state.json] for collection [deleteShard] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2167517 INFO  (zkCallback-7826-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteShard/state.json] for collection [deleteShard] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2167783 INFO  (OverseerCollectionConfigSetProcessor-72064765915955206-127.0.0.1:64189_solr-n_0000000000) [n:127.0.0.1:64189_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2168419 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf&name=deleteShard&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2640
   [junit4]   2> 2168421 INFO  (qtp1868790745-17504) [n:127.0.0.1:64184_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :deleteshard with params action=DELETESHARD&collection=deleteShard&shard=shard1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2168422 INFO  (OverseerThreadFactory-5998-thread-3-processing-n:127.0.0.1:64189_solr) [n:127.0.0.1:64189_solr c:deleteShard s:shard1  ] o.a.s.c.a.c.DeleteShardCmd Delete shard invoked
   [junit4]   2> 2168423 ERROR (OverseerThreadFactory-5998-thread-3-processing-n:127.0.0.1:64189_solr) [n:127.0.0.1:64189_solr c:deleteShard s:shard1  ] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: deleteShard operation: deleteshard failed:org.apache.solr.common.SolrException: The slice: shard1 is currently active. Only non-active (or custom-hashed) slices can be deleted.
   [junit4]   2> at org.apache.solr.cloud.api.collections.DeleteShardCmd.call(DeleteShardCmd.java:81)
   [junit4]   2> at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:259)
   [junit4]   2> at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:478)
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
   [junit4]   2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   [junit4]   2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   [junit4]   2> at java.base/java.lang.Thread.run(Thread.java:835)
   [junit4]   2>
   [junit4]   2> 2168425 INFO  (qtp1868790745-17504) [n:127.0.0.1:64184_solr c:deleteShard   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=DELETESHARD&collection=deleteShard&shard=shard1&wt=javabin&version=2} status=400 QTime=5
   [junit4]   2> 2168427 INFO  (OverseerStateUpdate-72064765915955206-127.0.0.1:64189_solr-n_0000000000) [n:127.0.0.1:64189_solr    ] o.a.s.c.o.SliceMutator Update shard state invoked for collection: deleteShard with message: {
   [junit4]   2>   "shard1":"inactive",
   [junit4]   2>   "collection":"deleteShard",
   [junit4]   2>   "operation":"updateshardstate"}
   [junit4]   2> 2168428 INFO  (OverseerStateUpdate-72064765915955206-127.0.0.1:64189_solr-n_0000000000) [n:127.0.0.1:64189_solr    ] o.a.s.c.o.SliceMutator Update shard state shard1 to inactive
   [junit4]   2> 2168529 INFO  (zkCallback-7826-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteShard/state.json] for collection [deleteShard] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2168529 INFO  (zkCallback-7818-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteShard/state.json] for collection [deleteShard] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2168529 INFO  (zkCallback-7847-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteShard/state.json] for collection [deleteShard] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2168530 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :deleteshard with params action=DELETESHARD&collection=deleteShard&shard=shard1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2168532 INFO  (OverseerThreadFactory-5998-thread-4-processing-n:127.0.0.1:64189_solr) [n:127.0.0.1:64189_solr c:deleteShard s:shard1  ] o.a.s.c.a.c.DeleteShardCmd Delete shard invoked
   [junit4]   2> 2168532 INFO  (OverseerCollectionConfigSetProcessor-72064765915955206-127.0.0.1:64189_solr-n_0000000000) [n:127.0.0.1:64189_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2168532 INFO  (OverseerThreadFactory-5998-thread-4-processing-n:127.0.0.1:64189_solr) [n:127.0.0.1:64189_solr c:deleteShard s:shard1  ] o.a.s.c.a.c.DeleteShardCmd Deleting replica for collection=deleteShard shard=shard1 on node=127.0.0.1:64189_solr
   [junit4]   2> 2168534 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr    x:deleteShard_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.deleteShard.shard1.replica_n1, tag=null
   [junit4]   2> 2168534 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr    x:deleteShard_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4ee15d34: rootName = solr_64189, domain = solr.core.deleteShard.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.deleteShard.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@54682b32
   [junit4]   2> 2168548 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr    x:deleteShard_shard1_replica_n1] o.a.s.c.SolrCore [deleteShard_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@690afcdb
   [junit4]   2> 2168548 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr    x:deleteShard_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.deleteShard.shard1.replica_n1, tag=690afcdb
   [junit4]   2> 2168548 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr    x:deleteShard_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.deleteShard.shard1.leader, tag=690afcdb
   [junit4]   2> 2168552 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr    x:deleteShard_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/deleteShard/terms/shard1 to Terms{values={}, version=1}
   [junit4]   2> 2168553 INFO  (qtp96379716-17511) [n:127.0.0.1:64189_solr    x:deleteShard_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=deleteShard_shard1_replica_n1&deleteMetricsHistory=true&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2&deleteIndex=true} status=0 QTime=20
   [junit4]   2> 2168656 INFO  (zkCallback-7826-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteShard/state.json] for collection [deleteShard] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2168656 INFO  (zkCallback-7818-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteShard/state.json] for collection [deleteShard] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2168758 INFO  (OverseerStateUpdate-72064765915955206-127.0.0.1:64189_solr-n_0000000000) [n:127.0.0.1:64189_solr    ] o.a.s.c.o.CollectionMutator Removing collection: deleteShard shard: shard1 from clusterstate
   [junit4]   2> 2168859 INFO  (zkCallback-7818-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteShard/state.json] for collection [deleteShard] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2169059 INFO  (OverseerThreadFactory-5998-thread-4-processing-n:127.0.0.1:64189_solr) [n:127.0.0.1:64189_solr c:deleteShard s:shard1  ] o.a.s.c.a.c.DeleteShardCmd Successfully deleted collection: deleteShard, shard: shard1
   [junit4]   2> 2169060 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr c:deleteShard   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=DELETESHARD&collection=deleteShard&shard=shard1&wt=javabin&version=2} status=0 QTime=530
   [junit4]   2> 2169064 INFO  (OverseerStateUpdate-72064765915955206-127.0.0.1:64189_solr-n_0000000000) [n:127.0.0.1:64189_solr    ] o.a.s.c.o.SliceMutator Update shard state invoked for collection: deleteShard with message: {
   [junit4]   2>   "shard2":"construction",
   [junit4]   2>   "collection":"deleteShard",
   [junit4]   2>   "operation":"updateshardstate"}
   [junit4]   2> 2169064 INFO  (OverseerStateUpdate-72064765915955206-127.0.0.1:64189_solr-n_0000000000) [n:127.0.0.1:64189_solr    ] o.a.s.c.o.SliceMutator Update shard state shard2 to construction
   [junit4]   2> 2169165 INFO  (zkCallback-7847-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteShard/state.json] for collection [deleteShard] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2169165 INFO  (zkCallback-7818-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteShard/state.json] for collection [deleteShard] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2169165 INFO  (zkCallback-7847-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteShard/state.json] for collection [deleteShard] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2169166 INFO  (qtp1868790745-17504) [n:127.0.0.1:64184_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :deleteshard with params action=DELETESHARD&collection=deleteShard&shard=shard2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2169167 INFO  (OverseerThreadFactory-5998-thread-5-processing-n:127.0.0.1:64189_solr) [n:127.0.0.1:64189_solr c:deleteShard s:shard2  ] o.a.s.c.a.c.DeleteShardCmd Delete shard invoked
   [junit4]   2> 2169168 INFO  (OverseerCollectionConfigSetProcessor-72064765915955206-127.0.0.1:64189_solr-n_0000000000) [n:127.0.0.1:64189_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2169168 INFO  (OverseerThreadFactory-5998-thread-5-processing-n:127.0.0.1:64189_solr) [n:127.0.0.1:64189_solr c:deleteShard s:shard2  ] o.a.s.c.a.c.DeleteShardCmd Deleting replica for collection=deleteShard shard=shard2 on node=127.0.0.1:64184_solr
   [junit4]   2> 2169168 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr    x:deleteShard_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.deleteShard.shard2.replica_n2, tag=null
   [junit4]   2> 2169168 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr    x:deleteShard_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@50aebbd7: rootName = solr_64184, domain = solr.core.deleteShard.shard2.replica_n2, service url = null, agent id = null] for registry solr.core.deleteShard.shard2.replica_n2 / com.codahale.metrics.MetricRegistry@4a3b3cd3
   [junit4]   2> 2169177 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr    x:deleteShard_shard2_replica_n2] o.a.s.c.SolrCore [deleteShard_shard2_replica_n2]  CLOSING SolrCore org.apache.solr.core.SolrCore@12a18feb
   [junit4]   2> 2169177 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr    x:deleteShard_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.deleteShard.shard2.replica_n2, tag=12a18feb
   [junit4]   2> 2169177 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr    x:deleteShard_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.deleteShard.shard2.leader, tag=12a18feb
   [junit4]   2> 2169182 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr    x:deleteShard_shard2_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/deleteShard/terms/shard2 to Terms{values={}, version=1}
   [junit4]   2> 2169183 INFO  (qtp1868790745-17501) [n:127.0.0.1:64184_solr    x:deleteShard_shard2_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=deleteShard_shard2_replica_n2&deleteMetricsHistory=true&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2&deleteIndex=true} status=0 QTime=14
   [junit4]   2> 2169285 INFO  (zkCallback-7818-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteShard/state.json] for collection [deleteShard] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2169388 INFO  (OverseerStateUpdate-72064765915955206-127.0.0.1:64189_solr-n_0000000000) [n:127.0.0.1:64189_solr    ] o.a.s.c.o.CollectionMutator Removing collection: deleteShard shard: shard2 from clusterstate
   [junit4]   2> 2169692 INFO  (OverseerThreadFactory-5998-thread-5-processing-n:127.0.0.1:64189_solr) [n:127.0.0.1:64189_solr c:deleteShard s:shard2  ] o.a.s.c.a.c.DeleteShardCmd Successfully deleted collection: deleteShard, shard: shard2
   [junit4]   2> 2169694 INFO  (qtp1868790745-17504) [n:127.0.0.1:64184_solr c:deleteShard   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=DELETESHARD&collection=deleteShard&shard=shard2&wt=javabin&version=2} status=0 QTime=528
   [junit4]   2> 2169695 INFO  (TEST-DeleteShardTest.test-seed#[8C6EDDB315F284EC]) [    ] o.a.s.SolrTestCaseJ4 ###Ending test
   [junit4]   2> 2169697 INFO  (jetty-closer-7807-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@b34e53b{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2169698 INFO  (jetty-closer-7807-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@7431dd78{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2169698 INFO  (jetty-closer-7807-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=414040944
   [junit4]   2> 2169698 INFO  (jetty-closer-7807-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2169698 INFO  (jetty-closer-7807-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1450960353
   [junit4]   2> 2169698 INFO  (jetty-closer-7807-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@63184538: rootName = solr_64184, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@600cfa8
   [junit4]   2> 2169699 INFO  (jetty-closer-7807-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2169699 INFO  (jetty-closer-7807-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1afe55a2: rootName = solr_64189, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@7eeee8ec
   [junit4]   2> 2169703 INFO  (jetty-closer-7807-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2169703 INFO  (jetty-closer-7807-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6d663866: rootName = solr_64184, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@692edeb9
   [junit4]   2> 2169706 INFO  (jetty-closer-7807-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2169706 INFO  (jetty-closer-7807-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2169706 INFO  (jetty-closer-7807-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6c14c03a: rootName = solr_64184, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@43a68cbd
   [junit4]   2> 2169706 INFO  (jetty-closer-7807-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5b6a59bb: rootName = solr_64189, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@692edeb9
   [junit4]   2> 2169708 INFO  (jetty-closer-7807-thread-1) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:64184_solr
   [junit4]   2> 2169709 INFO  (jetty-closer-7807-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 2169710 INFO  (zkCallback-7833-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 2169710 INFO  (zkCallback-7838-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 2169709 INFO  (zkCallback-7826-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 2169710 INFO  (zkCallback-7818-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 2169711 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1000685d53e0004, likely client has closed socket
   [junit4]   2> 2169711 INFO  (jetty-closer-7807-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2169711 INFO  (jetty-closer-7807-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7aad2637: rootName = solr_64189, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@43a68cbd
   [junit4]   2> 2169711 INFO  (jetty-closer-7807-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6b8f1854{/solr,null,UNAVAILABLE}
   [junit4]   2> 2169712 INFO  (jetty-closer-7807-thread-1) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 2169714 INFO  (jetty-closer-7807-thread-2) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:64189_solr
   [junit4]   2> 2169714 INFO  (jetty-closer-7807-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 2169715 INFO  (zkCallback-7833-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 2169715 INFO  (zkCallback-7826-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 2169716 INFO  (jetty

[...truncated too long message...]

s\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-interceptors-authn-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-interceptors-authz-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-interceptors-changelog-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-interceptors-collective-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-interceptors-event-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-interceptors-exception-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-interceptors-journal-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-interceptors-normalization-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-interceptors-operational-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-interceptors-referral-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-interceptors-schema-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-interceptors-subtree-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-interceptors-trigger-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-jdbm-partition-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-jdbm1-2.0.0-M2.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-kerberos-codec-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-ldif-partition-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-mavibot-partition-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-protocol-kerberos-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-protocol-ldap-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-protocol-shared-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\apacheds-xdbm-partition-2.0.0-M15.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\api-all-1.0.0-M20.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\bcprov-jdk15on-1.54.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\byte-buddy-1.6.2.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\commons-collections-3.2.2.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\ehcache-core-2.4.4.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\hadoop-common-2.7.4-tests.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\hadoop-hdfs-2.7.4-tests.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\hadoop-minikdc-2.7.4.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\htrace-core-3.2.0-incubating.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\jersey-core-1.9.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\jersey-server-1.9.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\jetty-6.1.26.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\jetty-sslengine-6.1.26.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\jetty-util-6.1.26.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\mina-core-2.0.0-M5.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\mockito-core-2.6.2.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\netty-all-4.0.36.Final.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\core\test-lib\objenesis-2.5.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\build\analysis\icu\lucene-analyzers-icu-7.6.0-SNAPSHOT.jar;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\contrib\solr-analysis-extras\classes\java;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\build\queryparser\classes\test;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\build\backward-codecs\classes\test;C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\contrib\analysis-extras\lib\icu4j-62.1.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-launcher.jar;C:\Users\jenkins\.ant\lib\ivy-2.4.0.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-antlr.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-apache-bcel.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-apache-bsf.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-apache-log4j.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-apache-oro.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-apache-regexp.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-apache-resolver.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-apache-xalan2.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-commons-logging.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-commons-net.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-jai.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-javamail.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-jdepend.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-jmf.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-jsch.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-junit.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-junit4.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-netrexx.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-swing.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant-testutil.jar;C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\lib\ant.jar;C:\Users\jenkins\.ivy2\cache\com.carrotsearch.randomizedtesting\junit4-ant\jars\junit4-ant-2.6.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\temp\junit4-J0-20181011_020404_29713643886912528975344.events @C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\temp\junit4-J0-20181011_020404_2977552896555968632123.suites -stdin
   [junit4] ERROR: JVM J0 ended with an exception: Forked process returned with error code: 1. Very likely a JVM crash.  See process stdout at: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\temp\junit4-J0-20181011_020404_29711199522383324662526.sysout
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1542)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:123)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:997)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:994)
   [junit4] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   [junit4] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   [junit4] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   [junit4] at java.base/java.lang.Thread.run(Thread.java:835)

BUILD FAILED
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\build.xml:633: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\build.xml:577: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\build.xml:59: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build.xml:267: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\common-build.xml:558: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\common-build.xml:1567: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\common-build.xml:1092: At least one slave process threw an exception, first: Forked process returned with error code: 1. Very likely a JVM crash.  See process stdout at: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\temp\junit4-J0-20181011_020404_29711199522383324662526.sysout

Total time: 97 minutes 59 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]