[JENKINS] Lucene-Solr-BadApples-Tests-7.x - Build # 38 - Failure

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-Tests-7.x - Build # 38 - Failure

Apache Jenkins Server-2
Error processing tokens: Error while parsing action 'Text/ZeroOrMore/FirstOf/Token/DelimitedToken/DelimitedToken_Action3' at input position (line 76, pos 4):
)"}
   ^

java.lang.OutOfMemoryError: Java heap space


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-Tests-7.x - Build # 39 - Still Failing

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/39/

3 tests failed.
FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([D93B52828EC50BB8]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.HttpPartitionTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([D93B52828EC50BB8]:0)


FAILED:  org.apache.solr.cloud.MoveReplicaHDFSTest.testFailedMove

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:43008/solr/MoveReplicaHDFSTest_failed_coll_true, http://127.0.0.1:37177/solr/MoveReplicaHDFSTest_failed_coll_true, http://127.0.0.1:42851/solr/MoveReplicaHDFSTest_failed_coll_true]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:43008/solr/MoveReplicaHDFSTest_failed_coll_true, http://127.0.0.1:37177/solr/MoveReplicaHDFSTest_failed_coll_true, http://127.0.0.1:42851/solr/MoveReplicaHDFSTest_failed_coll_true]
        at __randomizedtesting.SeedInfo.seed([D93B52828EC50BB8:73F681703916DE68]:0)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:462)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1106)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:993)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.cloud.MoveReplicaTest.testFailedMove(MoveReplicaTest.java:308)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:42851/solr/MoveReplicaHDFSTest_failed_coll_true: Error from server at http://127.0.0.1:37177/solr/MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2/select</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2/select. Reason:
<pre>    Can not find: /solr/MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2/select</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.8.v20171121</a><hr/>

</body>
</html>

        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        ... 46 more




Build Log:
[...truncated 13766 lines...]
   [junit4] Suite: org.apache.solr.cloud.MoveReplicaHDFSTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_D93B52828EC50BB8-001/init-core-data-001
   [junit4]   2> 2846977 WARN  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=6 numCloses=6
   [junit4]   2> 2846977 INFO  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2846978 INFO  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2846978 INFO  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2846979 INFO  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_D93B52828EC50BB8-001/tempDir-001
   [junit4]   2> 2846979 INFO  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2846979 INFO  (Thread-4640) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2846979 INFO  (Thread-4640) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2846983 ERROR (Thread-4640) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2847079 INFO  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:42248
   [junit4]   2> 2847084 INFO  (zkConnectionManagerCallback-7323-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2847091 INFO  (jetty-launcher-7320-thread-3) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2847091 INFO  (jetty-launcher-7320-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2847091 INFO  (jetty-launcher-7320-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2847091 INFO  (jetty-launcher-7320-thread-4) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2847092 INFO  (jetty-launcher-7320-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2847092 INFO  (jetty-launcher-7320-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2847092 INFO  (jetty-launcher-7320-thread-2) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 2847093 INFO  (jetty-launcher-7320-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2847093 INFO  (jetty-launcher-7320-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2847093 INFO  (jetty-launcher-7320-thread-3) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 2847093 INFO  (jetty-launcher-7320-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2da18c21{/solr,null,AVAILABLE}
   [junit4]   2> 2847093 INFO  (jetty-launcher-7320-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2847093 INFO  (jetty-launcher-7320-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2847093 INFO  (jetty-launcher-7320-thread-1) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 2847093 INFO  (jetty-launcher-7320-thread-4) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2847093 INFO  (jetty-launcher-7320-thread-4) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2847093 INFO  (jetty-launcher-7320-thread-4) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 2847093 INFO  (jetty-launcher-7320-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2eb37e71{/solr,null,AVAILABLE}
   [junit4]   2> 2847094 INFO  (jetty-launcher-7320-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4d09c515{HTTP/1.1,[http/1.1]}{127.0.0.1:39928}
   [junit4]   2> 2847094 INFO  (jetty-launcher-7320-thread-2) [    ] o.e.j.s.Server Started @2847210ms
   [junit4]   2> 2847094 INFO  (jetty-launcher-7320-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=39928}
   [junit4]   2> 2847094 ERROR (jetty-launcher-7320-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2847094 INFO  (jetty-launcher-7320-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2847094 INFO  (jetty-launcher-7320-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.4.0
   [junit4]   2> 2847094 INFO  (jetty-launcher-7320-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2847094 INFO  (jetty-launcher-7320-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2847094 INFO  (jetty-launcher-7320-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-04-12T14:37:17.495Z
   [junit4]   2> 2847095 INFO  (jetty-launcher-7320-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@547c78a7{/solr,null,AVAILABLE}
   [junit4]   2> 2847095 INFO  (jetty-launcher-7320-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@104449f3{/solr,null,AVAILABLE}
   [junit4]   2> 2847095 INFO  (jetty-launcher-7320-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5942d19{HTTP/1.1,[http/1.1]}{127.0.0.1:37177}
   [junit4]   2> 2847096 INFO  (jetty-launcher-7320-thread-3) [    ] o.e.j.s.Server Started @2847211ms
   [junit4]   2> 2847096 INFO  (jetty-launcher-7320-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37177}
   [junit4]   2> 2847096 ERROR (jetty-launcher-7320-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2847096 INFO  (jetty-launcher-7320-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2847096 INFO  (jetty-launcher-7320-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.4.0
   [junit4]   2> 2847096 INFO  (jetty-launcher-7320-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2847096 INFO  (jetty-launcher-7320-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2847096 INFO  (jetty-launcher-7320-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-04-12T14:37:17.497Z
   [junit4]   2> 2847097 INFO  (jetty-launcher-7320-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@41108655{HTTP/1.1,[http/1.1]}{127.0.0.1:43887}
   [junit4]   2> 2847097 INFO  (jetty-launcher-7320-thread-1) [    ] o.e.j.s.Server Started @2847212ms
   [junit4]   2> 2847097 INFO  (jetty-launcher-7320-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43887}
   [junit4]   2> 2847097 ERROR (jetty-launcher-7320-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2847097 INFO  (jetty-launcher-7320-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2847097 INFO  (jetty-launcher-7320-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.4.0
   [junit4]   2> 2847097 INFO  (jetty-launcher-7320-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2847097 INFO  (jetty-launcher-7320-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2847097 INFO  (jetty-launcher-7320-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-04-12T14:37:17.498Z
   [junit4]   2> 2847098 INFO  (jetty-launcher-7320-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2d587425{HTTP/1.1,[http/1.1]}{127.0.0.1:42851}
   [junit4]   2> 2847098 INFO  (jetty-launcher-7320-thread-4) [    ] o.e.j.s.Server Started @2847213ms
   [junit4]   2> 2847098 INFO  (jetty-launcher-7320-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42851}
   [junit4]   2> 2847098 ERROR (jetty-launcher-7320-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2847098 INFO  (jetty-launcher-7320-thread-4) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2847098 INFO  (zkConnectionManagerCallback-7325-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2847098 INFO  (jetty-launcher-7320-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.4.0
   [junit4]   2> 2847098 INFO  (jetty-launcher-7320-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2847098 INFO  (jetty-launcher-7320-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2847098 INFO  (jetty-launcher-7320-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-04-12T14:37:17.499Z
   [junit4]   2> 2847098 INFO  (zkConnectionManagerCallback-7329-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2847099 INFO  (jetty-launcher-7320-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2847141 INFO  (zkConnectionManagerCallback-7327-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2847141 INFO  (zkConnectionManagerCallback-7331-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2847142 INFO  (jetty-launcher-7320-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2847142 INFO  (jetty-launcher-7320-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2847142 INFO  (jetty-launcher-7320-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2847156 INFO  (jetty-launcher-7320-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42248/solr
   [junit4]   2> 2847159 INFO  (jetty-launcher-7320-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42248/solr
   [junit4]   2> 2847159 INFO  (jetty-launcher-7320-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42248/solr
   [junit4]   2> 2847162 INFO  (zkConnectionManagerCallback-7335-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2847172 INFO  (zkConnectionManagerCallback-7343-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2847172 INFO  (zkConnectionManagerCallback-7341-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2847172 INFO  (jetty-launcher-7320-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42248/solr
   [junit4]   2> 2847176 INFO  (zkConnectionManagerCallback-7346-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2847176 INFO  (zkConnectionManagerCallback-7349-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2847177 INFO  (zkConnectionManagerCallback-7351-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2847178 INFO  (zkConnectionManagerCallback-7353-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2847179 INFO  (zkConnectionManagerCallback-7355-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2847230 INFO  (jetty-launcher-7320-thread-1) [n:127.0.0.1:43887_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2847231 INFO  (jetty-launcher-7320-thread-1) [n:127.0.0.1:43887_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:43887_solr
   [junit4]   2> 2847232 INFO  (jetty-launcher-7320-thread-1) [n:127.0.0.1:43887_solr    ] o.a.s.c.Overseer Overseer (id=72443026872336396-127.0.0.1:43887_solr-n_0000000000) starting
   [junit4]   2> 2847262 INFO  (zkConnectionManagerCallback-7368-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2847269 INFO  (jetty-launcher-7320-thread-1) [n:127.0.0.1:43887_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42248/solr ready
   [junit4]   2> 2847270 INFO  (jetty-launcher-7320-thread-1) [n:127.0.0.1:43887_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43887_solr
   [junit4]   2> 2847306 INFO  (OverseerStateUpdate-72443026872336396-127.0.0.1:43887_solr-n_0000000000) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2847352 INFO  (zkCallback-7367-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2847394 INFO  (jetty-launcher-7320-thread-4) [n:127.0.0.1:42851_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2847396 INFO  (jetty-launcher-7320-thread-3) [n:127.0.0.1:37177_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2847397 INFO  (jetty-launcher-7320-thread-4) [n:127.0.0.1:42851_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2847399 INFO  (jetty-launcher-7320-thread-3) [n:127.0.0.1:37177_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2847408 INFO  (jetty-launcher-7320-thread-4) [n:127.0.0.1:42851_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2847408 INFO  (jetty-launcher-7320-thread-4) [n:127.0.0.1:42851_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42851_solr
   [junit4]   2> 2847409 INFO  (zkCallback-7367-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2847410 INFO  (zkCallback-7354-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2847418 INFO  (jetty-launcher-7320-thread-2) [n:127.0.0.1:39928_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2847419 INFO  (jetty-launcher-7320-thread-2) [n:127.0.0.1:39928_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2847421 INFO  (jetty-launcher-7320-thread-2) [n:127.0.0.1:39928_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2847421 INFO  (jetty-launcher-7320-thread-2) [n:127.0.0.1:39928_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39928_solr
   [junit4]   2> 2847435 INFO  (zkCallback-7367-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2847448 INFO  (zkCallback-7352-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 2847450 INFO  (zkCallback-7354-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2847466 INFO  (zkCallback-7345-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2847485 INFO  (jetty-launcher-7320-thread-3) [n:127.0.0.1:37177_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2847485 INFO  (jetty-launcher-7320-thread-3) [n:127.0.0.1:37177_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37177_solr
   [junit4]   2> 2847487 INFO  (zkCallback-7345-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2847487 INFO  (zkCallback-7354-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2847519 INFO  (zkCallback-7350-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 2847549 INFO  (zkCallback-7367-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2847554 INFO  (zkCallback-7352-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2847554 INFO  (zkCallback-7350-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2847767 INFO  (jetty-launcher-7320-thread-1) [n:127.0.0.1:43887_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43887.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2847773 INFO  (jetty-launcher-7320-thread-2) [n:127.0.0.1:39928_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39928.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2847780 INFO  (jetty-launcher-7320-thread-2) [n:127.0.0.1:39928_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39928.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2847780 INFO  (jetty-launcher-7320-thread-2) [n:127.0.0.1:39928_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39928.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2847781 INFO  (jetty-launcher-7320-thread-2) [n:127.0.0.1:39928_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_D93B52828EC50BB8-001/tempDir-001/node2/.
   [junit4]   2> 2847783 INFO  (jetty-launcher-7320-thread-1) [n:127.0.0.1:43887_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43887.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2847783 INFO  (jetty-launcher-7320-thread-1) [n:127.0.0.1:43887_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43887.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2847784 INFO  (jetty-launcher-7320-thread-1) [n:127.0.0.1:43887_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_D93B52828EC50BB8-001/tempDir-001/node1/.
   [junit4]   2> 2847792 INFO  (zkConnectionManagerCallback-7373-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2847794 INFO  (jetty-launcher-7320-thread-2) [n:127.0.0.1:39928_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2847795 INFO  (jetty-launcher-7320-thread-2) [n:127.0.0.1:39928_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42248/solr ready
   [junit4]   2> 2847858 INFO  (jetty-launcher-7320-thread-4) [n:127.0.0.1:42851_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42851.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2847865 INFO  (jetty-launcher-7320-thread-4) [n:127.0.0.1:42851_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42851.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2847865 INFO  (jetty-launcher-7320-thread-4) [n:127.0.0.1:42851_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42851.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2847866 INFO  (jetty-launcher-7320-thread-4) [n:127.0.0.1:42851_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_D93B52828EC50BB8-001/tempDir-001/node4/.
   [junit4]   2> 2847869 INFO  (zkConnectionManagerCallback-7378-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2847870 INFO  (jetty-launcher-7320-thread-4) [n:127.0.0.1:42851_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2847870 INFO  (jetty-launcher-7320-thread-4) [n:127.0.0.1:42851_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42248/solr ready
   [junit4]   2> 2848004 INFO  (jetty-launcher-7320-thread-3) [n:127.0.0.1:37177_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37177.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2848011 INFO  (jetty-launcher-7320-thread-3) [n:127.0.0.1:37177_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37177.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2848011 INFO  (jetty-launcher-7320-thread-3) [n:127.0.0.1:37177_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37177.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2848012 INFO  (jetty-launcher-7320-thread-3) [n:127.0.0.1:37177_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_D93B52828EC50BB8-001/tempDir-001/node3/.
   [junit4]   2> 2848015 INFO  (zkConnectionManagerCallback-7383-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2848016 INFO  (jetty-launcher-7320-thread-3) [n:127.0.0.1:37177_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2848017 INFO  (jetty-launcher-7320-thread-3) [n:127.0.0.1:37177_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42248/solr ready
   [junit4]   2> 2848064 INFO  (zkConnectionManagerCallback-7385-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2848067 INFO  (zkConnectionManagerCallback-7390-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2848068 INFO  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2848068 INFO  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42248/solr ready
   [junit4]   2> 2848072 INFO  (qtp2003698812-22787) [n:127.0.0.1:43887_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2848100 INFO  (qtp2003698812-22787) [n:127.0.0.1:43887_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=28
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 2848181 WARN  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 2848207 WARN  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2848208 INFO  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2848240 INFO  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_lucene2.us.west_apache_org_34931_hdfs____5whc5c/webapp
   [junit4]   2> 2848730 INFO  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.m.log Started HttpServer2$[hidden email]:34931
   [junit4]   2> 2849433 WARN  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2849434 INFO  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2849542 INFO  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_33839_datanode____p3mbs5/webapp
   [junit4]   2> 2850114 INFO  (OverseerCollectionConfigSetProcessor-72443026872336396-127.0.0.1:43887_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2850595 INFO  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33839
   [junit4]   2> 2851459 WARN  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2851464 INFO  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2851602 INFO  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_35699_datanode____.ksa1np/webapp
   [junit4]   2> 2852079 ERROR (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_D93B52828EC50BB8-001/tempDir-002/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_D93B52828EC50BB8-001/tempDir-002/hdfsBaseDir/data/data2/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:46819) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 2852185 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x14e4f5b6edf74f: from storage DS-eb984553-996c-4b9b-b087-af607130edec node DatanodeRegistration(127.0.0.1:34546, datanodeUuid=6b7471b1-d0bc-4eea-b1b9-b210f1e68c0a, infoPort=37266, infoSecurePort=0, ipcPort=41411, storageInfo=lv=-56;cid=testClusterID;nsid=1825728911;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 2852185 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x14e4f5b6edf74f: from storage DS-4bf5ee87-9bf2-4247-81f0-6922a233962a node DatanodeRegistration(127.0.0.1:34546, datanodeUuid=6b7471b1-d0bc-4eea-b1b9-b210f1e68c0a, infoPort=37266, infoSecurePort=0, ipcPort=41411, storageInfo=lv=-56;cid=testClusterID;nsid=1825728911;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 2852546 INFO  (SUITE-MoveReplicaHDFSTest-seed#[D93B52828EC50BB8]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35699
   [junit4]   2> 2852763 ERROR (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_D93B52828EC50BB8-001/tempDir-002/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_D93B52828EC50BB8-001/tempDir-002/hdfsBaseDir/data/data4/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:46819) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 2852792 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x14e4f5db36aa1f: from storage DS-9dafaf1f-cce1-4816-acff-7f3923c3c568 node DatanodeRegistration(127.0.0.1:40655, datanodeUuid=59ea0808-3440-4e98-abda-4cbb700557dc, infoPort=37348, infoSecurePort=0, ipcPort=41828, storageInfo=lv=-56;cid=testClusterID;nsid=1825728911;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 2852792 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x14e4f5db36aa1f: from storage DS-14749d85-1b31-40d5-a4e6-eba358d92531 node DatanodeRegistration(127.0.0.1:40655, datanodeUuid=59ea0808-3440-4e98-abda-4cbb700557dc, infoPort=37348, infoSecurePort=0, ipcPort=41828, storageInfo=lv=-56;cid=testClusterID;nsid=1825728911;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 2853174 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testFailedMove
   [junit4]   2> 2853175 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2853176 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2853177 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2853177 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2853177 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 2853177 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@c2f02ff{/solr,null,AVAILABLE}
   [junit4]   2> 2853178 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@11324d3c{HTTP/1.1,[http/1.1]}{127.0.0.1:43008}
   [junit4]   2> 2853178 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.e.j.s.Server Started @2853294ms
   [junit4]   2> 2853178 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43008}
   [junit4]   2> 2853178 ERROR (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2853178 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2853178 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.4.0
   [junit4]   2> 2853178 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2853178 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2853178 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-04-12T14:37:23.579Z
   [junit4]   2> 2853180 INFO  (zkConnectionManagerCallback-7394-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2853180 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2853200 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42248/solr
   [junit4]   2> 2853201 INFO  (zkConnectionManagerCallback-7398-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2853203 INFO  (zkConnectionManagerCallback-7400-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2853208 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [n:127.0.0.1:43008_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2853210 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [n:127.0.0.1:43008_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2853211 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [n:127.0.0.1:43008_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2853211 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [n:127.0.0.1:43008_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43008_solr
   [junit4]   2> 2853213 INFO  (zkCallback-7352-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2853213 INFO  (zkCallback-7354-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2853213 INFO  (zkCallback-7345-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2853213 INFO  (zkCallback-7367-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2853248 INFO  (zkCallback-7377-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2853248 INFO  (zkCallback-7350-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2853248 INFO  (zkCallback-7399-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2853248 INFO  (zkCallback-7372-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2853256 INFO  (zkCallback-7389-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2853257 INFO  (zkCallback-7389-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2853257 INFO  (zkCallback-7382-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2853516 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [n:127.0.0.1:43008_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43008.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2853525 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [n:127.0.0.1:43008_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43008.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2853525 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [n:127.0.0.1:43008_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43008.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2853527 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [n:127.0.0.1:43008_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_D93B52828EC50BB8-001/tempDir-001/node5/.
   [junit4]   2> 2853537 INFO  (zkConnectionManagerCallback-7407-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2853538 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [n:127.0.0.1:43008_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 2853539 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[D93B52828EC50BB8]) [n:127.0.0.1:43008_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42248/solr ready
   [junit4]   2> 2855174 INFO  (zkConnectionManagerCallback-7409-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2855177 INFO  (qtp1603064068-22788) [n:127.0.0.1:37177_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf1&autoAddReplicas=false&name=MoveReplicaHDFSTest_failed_coll_true&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2855179 INFO  (OverseerThreadFactory-8209-thread-2) [    ] o.a.s.c.a.c.CreateCollectionCmd Create collection MoveReplicaHDFSTest_failed_coll_true
   [junit4]   2> 2855290 INFO  (OverseerStateUpdate-72443026872336396-127.0.0.1:43887_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"MoveReplicaHDFSTest_failed_coll_true",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:43887/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2855292 INFO  (OverseerStateUpdate-72443026872336396-127.0.0.1:43887_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"MoveReplicaHDFSTest_failed_coll_true",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:37177/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2855294 INFO  (OverseerStateUpdate-72443026872336396-127.0.0.1:43887_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"MoveReplicaHDFSTest_failed_coll_true",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:42851/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2855309 INFO  (OverseerStateUpdate-72443026872336396-127.0.0.1:43887_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"MoveReplicaHDFSTest_failed_coll_true",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:43008/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2855511 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf1&newCollection=true&name=MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1&action=CREATE&numShards=2&collection=MoveReplicaHDFSTest_failed_coll_true&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2855511 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf1&newCollection=true&name=MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2&action=CREATE&numShards=2&collection=MoveReplicaHDFSTest_failed_coll_true&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2855512 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2855515 INFO  (qtp1272766324-22800) [n:127.0.0.1:42851_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf1&newCollection=true&name=MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4&action=CREATE&numShards=2&collection=MoveReplicaHDFSTest_failed_coll_true&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2855515 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf1&newCollection=true&name=MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6&action=CREATE&numShards=2&collection=MoveReplicaHDFSTest_failed_coll_true&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2855640 INFO  (zkCallback-7399-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 2855640 INFO  (zkCallback-7354-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 2855640 INFO  (zkCallback-7350-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 2855640 INFO  (zkCallback-7354-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 2855640 INFO  (zkCallback-7352-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 2855640 INFO  (zkCallback-7350-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 2855640 INFO  (zkCallback-7399-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 2855641 INFO  (zkCallback-7352-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 2856547 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 2856547 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 2856570 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 2856570 INFO  (qtp1272766324-22800) [n:127.0.0.1:42851_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node7 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 2856575 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.s.IndexSchema [MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] Schema name=minimal
   [junit4]   2> 2856577 INFO  (qtp1272766324-22800) [n:127.0.0.1:42851_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node7 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4] o.a.s.s.IndexSchema [MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4] Schema name=minimal
   [junit4]   2> 2856589 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2856589 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6' using configuration from collection MoveReplicaHDFSTest_failed_coll_true, trusted=true
   [junit4]   2> 2856590 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43008.solr.core.MoveReplicaHDFSTest_failed_coll_true.shard2.replica_n6' (registry 'solr.core.MoveReplicaHDFSTest_failed_coll_true.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2856590 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:46819/data
   [junit4]   2> 2856590 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 2856590 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2856590 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.c.SolrCore [[MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_D93B52828EC50BB8-001/tempDir-001/node5/MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6], dataDir=[hdfs://lucene2-us-west.apache.org:46819/data/MoveReplicaHDFSTest_failed_coll_true/core_node8/data/]
   [junit4]   2> 2856592 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46819/data/MoveReplicaHDFSTest_failed_coll_true/core_node8/data/snapshot_metadata
   [junit4]   2> 2856599 INFO  (qtp1272766324-22800) [n:127.0.0.1:42851_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node7 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2856599 INFO  (qtp1272766324-22800) [n:127.0.0.1:42851_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node7 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4' using configuration from collection MoveReplicaHDFSTest_failed_coll_true, trusted=true
   [junit4]   2> 2856599 INFO  (qtp1272766324-22800) [n:127.0.0.1:42851_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node7 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42851.solr.core.MoveReplicaHDFSTest_failed_coll_true.shard2.replica_n4' (registry 'solr.core.MoveReplicaHDFSTest_failed_coll_true.shard2.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2856600 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.s.IndexSchema [MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 2856600 INFO  (qtp1272766324-22800) [n:127.0.0.1:42851_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node7 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:46819/data
   [junit4]   2> 2856600 INFO  (qtp1272766324-22800) [n:127.0.0.1:42851_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node7 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 2856600 INFO  (qtp1272766324-22800) [n:127.0.0.1:42851_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node7 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2856600 INFO  (qtp1272766324-22800) [n:127.0.0.1:42851_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node7 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4] o.a.s.c.SolrCore [[MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_D93B52828EC50BB8-001/tempDir-001/node4/MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4], dataDir=[hdfs://lucene2-us-west.apache.org:46819/data/MoveReplicaHDFSTest_failed_coll_true/core_node7/data/]
   [junit4]   2> 2856601 INFO  (qtp1272766324-22800) [n:127.0.0.1:42851_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node7 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46819/data/MoveReplicaHDFSTest_failed_coll_true/core_node7/data/snapshot_metadata
   [junit4]   2> 2856601 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2856601 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2' using configuration from collection MoveReplicaHDFSTest_failed_coll_true, trusted=true
   [junit4]   2> 2856615 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.s.IndexSchema [MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 2856665 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37177.solr.core.MoveReplicaHDFSTest_failed_coll_true.shard1.replica_n2' (registry 'solr.core.MoveReplicaHDFSTest_failed_coll_true.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2856665 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:46819/data
   [junit4]   2> 2856665 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 2856666 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2856666 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.c.SolrCore [[MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_D93B52828EC50BB8-001/tempDir-001/node3/MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2], dataDir=[hdfs://lucene2-us-west.apache.org:46819/data/MoveReplicaHDFSTest_failed_coll_true/core_node5/data/]
   [junit4]   2> 2856675 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46819/data/MoveReplicaHDFSTest_failed_coll_true/core_node5/data/snapshot_metadata
   [junit4]   2> 2856679 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2856679 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1' using configuration from collection MoveReplicaHDFSTest_failed_coll_true, trusted=true
   [junit4]   2> 2856680 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43887.solr.core.MoveReplicaHDFSTest_failed_coll_true.shard1.replica_n1' (registry 'solr.core.MoveReplicaHDFSTest_failed_coll_true.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11504f69
   [junit4]   2> 2856680 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:46819/data
   [junit4]   2> 2856680 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 2856680 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2856680 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.SolrCore [[MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_D93B52828EC50BB8-001/tempDir-001/node1/MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1], dataDir=[hdfs://lucene2-us-west.apache.org:46819/data/MoveReplicaHDFSTest_failed_coll_true/core_node3/data/]
   [junit4]   2> 2856681 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46819/data/MoveReplicaHDFSTest_failed_coll_true/core_node3/data/snapshot_metadata
   [junit4]   2> 2856727 WARN  (qtp1272766324-22800) [n:127.0.0.1:42851_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node7 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 2856742 WARN  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 2856742 WARN  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 2856746 WARN  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 2862163 INFO  (qtp1272766324-22800) [n:127.0.0.1:42851_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node7 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46819/data/MoveReplicaHDFSTest_failed_coll_true/core_node7/data
   [junit4]   2> 2862188 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46819/data/MoveReplicaHDFSTest_failed_coll_true/core_node3/data
   [junit4]   2> 2862195 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46819/data/MoveReplicaHDFSTest_failed_coll_true/core_node8/data
   [junit4]   2> 2862201 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46819/data/MoveReplicaHDFSTest_failed_coll_true/core_node5/data
   [junit4]   2> 2862261 INFO  (qtp1272766324-22800) [n:127.0.0.1:42851_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node7 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n4] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46819/data/MoveReplicaHDFSTest_failed_coll_true/core_node7/data/index
   [junit4]   2> 2862286 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46819/data/MoveReplicaHDFSTest_failed_coll_true/core_node5/data/index
   [junit4]   2> 2862286 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46819/data/MoveReplicaHDFSTest_failed_coll_true/core_node3/data/index
   [junit4]   2> 2862304 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46819/data/MoveReplicaHDFSTest_failed_coll_true/core_node8/data/index
   [junit4]   2> 2862676 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:40655 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-eb984553-996c-4b9b-b087-af607130edec:NORMAL:127.0.0.1:34546|RBW], ReplicaUC[[DISK]DS-14749d85-1b31-40d5-a4e6-eba358d92531:NORMAL:127.0.0.1:40655|RBW]]} size 0
   [junit4]   2> 2862676 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:40655 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-eb984553-996c-4b9b-b087-af607130edec:NORMAL:127.0.0.1:34546|RBW], ReplicaUC[[DISK]DS-9dafaf1f-cce1-4816-acff-7f3923c3c568:NORMAL:127.0.0.1:40655|RBW]]} size 0
   [junit4]   2> 2862690 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:34546 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-eb984553-996c-4b9b-b087-af607130edec:NORMAL:127.0.0.1:34546|RBW], ReplicaUC[[DISK]DS-9dafaf1f-cce1-4816-acff-7f3923c3c568:NORMAL:127.0.0.1:40655|RBW]]} size 0
   [junit4]   2> 2862718 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:40655 is added to blk_1073741828_1004{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-14749d85-1b31-40d5-a4e6-eba358d92531:NORMAL:127.0.0.1:40655|RBW], ReplicaUC[[DISK]DS-eb984553-996c-4b9b-b087-af607130edec:NORMAL:127.0.0.1:34546|RBW]]} size 69
   [junit4]   2> 2862718 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:34546 is added to blk_1073741828_1004 size 69
   [junit4]   2> 2862718 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:34546 is added to blk_1073741826_1002 size 69
   [junit4]   2> 2862719 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:34546 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9dafaf1f-cce1-4816-acff-7f3923c3c568:NORMAL:127.0.0.1:40655|RBW], ReplicaUC[[DISK]DS-4bf5ee87-9bf2-4247-81f0-6922a233962a:NORMAL:127.0.0.1:34546|RBW]]} size 0
   [junit4]   2> 2862721 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:40655 is added to blk_1073741827_1003 size 69
   [junit4]   2> 2862975 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 2862975 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 2862975 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2862975 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2862975 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 2862975 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 2863019 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2863019 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2863033 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 2863033 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2863033 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 2863039 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2863039 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2863145 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2863145 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2863187 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.s.SolrIndexSearcher Opening [Searcher@16e43fe2[MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] main]
   [junit4]   2> 2863205 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2863205 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2863206 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2863210 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@a3353de[MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] main]
   [junit4]   2> 2863211 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2863211 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2863228 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2863238 INFO  (searcherExecutor-8227-thread-1-processing-n:127.0.0.1:43887_solr x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1 c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.SolrCore [MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] Registered new searcher Searcher@a3353de[MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2863238 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1597551519873368064
   [junit4]   2> 2863248 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1597551519883853824
   [junit4]   2> 2863252 INFO  (searcherExecutor-8224-thread-1-processing-n:127.0.0.1:43008_solr x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6 c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.c.SolrCore [MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] Registered new searcher Searcher@16e43fe2[MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2863288 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/MoveReplicaHDFSTest_failed_coll_true/terms/shard2 to Terms{values={core_node8=0}, version=0}
   [junit4]   2> 2863304 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@b4f36b9[MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] main]
   [junit4]   2> 2863305 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2863306 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2863306 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2863307 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1597551519945719808
   [junit4]   2> 2863321 INFO  (searcherExecutor-8226-thread-1-processing-n:127.0.0.1:37177_solr x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2 c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.c.SolrCore [MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] Registered new searcher Searcher@b4f36b9[MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2863323 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/MoveReplicaHDFSTest_failed_coll_true/terms/shard1 to Terms{values={core_node5=0}, version=0}
   [junit4]   2> 2863342 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.ZkShardTerms Failed to save terms, version is not a match, retrying
   [junit4]   2> 2863344 INFO  (qtp2003698812-22801) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/MoveReplicaHDFSTest_failed_coll_true/terms/shard1 to Terms{values={core_node3=0, core_node5=0}, version=1}
   [junit4]   2> 2863379 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2863379 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2863379 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:37177/solr/MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2/
   [junit4]   2> 2863379 INFO  (qtp1603064068-22806) [n:127.0.0.1:37177_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node5 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n2 url=http://127.0.0.1:37177/solr START replicas=[http://127.0.0.1:43887/solr/MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1/] nUpdates=100
   [junit4]   2> 2863380 INFO  (qtp1737811792-23071) [n:127.0.0.1:43008_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 2863402 INFO  (zkCallback-7399-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 2863402 INFO  (zkCallback-7399-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 2863418 INFO  (zkCallback-7354-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 2863418 INFO  (zkCallback-7354-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 2863418 INFO  (qtp2003698812-22810) [n:127.0.0.1:43887_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node3 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.S.Request [MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=35
   [junit4]   2> 2863418 INFO  (zkCallback-7350-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 2863419 INFO  (zkCallback-7352-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 2863419 INFO  (zkCallback-7350-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 2863419 INFO  (zkCallback-7352-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size

[...truncated too long message...]

oreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>   19) Thread[id=4674, name=zkConnectionManagerCallback-1007-thread-1, state=WAITING, group=TGRP-HttpPartitionTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>   20) Thread[id=4618, name=Scheduler-1483861829, state=TIMED_WAITING, group=TGRP-HttpPartitionTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]   2>         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   [junit4]   2>         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>   21) Thread[id=4679, name=Scheduler-96649350, state=WAITING, group=TGRP-HttpPartitionTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
   [junit4]   2>         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>   22) Thread[id=4636, name=zkConnectionManagerCallback-992-thread-1, state=WAITING, group=TGRP-HttpPartitionTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>   23) Thread[id=4615, name=qtp1806234425-4615, state=RUNNABLE, group=TGRP-HttpPartitionTest]
   [junit4]   2>         at sun.nio.ch.IOUtil.drain(Native Method)
   [junit4]   2>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:104)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   [junit4]   2>         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)
   [junit4]   2>         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)
   [junit4]   2>         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)
   [junit4]   2>         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
   [junit4]   2>         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
   [junit4]   2>         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
   [junit4]   2>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]   2>         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=489, maxMBSortInHeap=6.10878187238618, sim=RandomSimilarity(queryNorm=false): {}, locale=mt, timezone=US/Pacific-New
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=43,free=264313616,total=472383488
   [junit4]   2> NOTE: All tests run in this JVM: [FullSolrCloudDistribCmdsTest, TestHdfsUpdateLog, TestSolrCoreSnapshots, ChaosMonkeyNothingIsSafeTest, TestEmbeddedSolrServerConstructors, ComputePlanActionTest, TestCoreBackup, HighlighterConfigTest, TestFastOutputStream, TestLegacyFieldCache, TestDocSet, TestTriggerIntegration, TestPhraseSuggestions, TestUninvertingReader, DistributedFacetExistsSmallTest, TestBinaryResponseWriter, RAMDirectoryFactoryTest, TestComponentsName, HdfsThreadLeakTest, SolrPluginUtilsTest, TestUseDocValuesAsStored2, HdfsDirectoryFactoryTest, DebugComponentTest, TestMergePolicyConfig, TestSolrQueryParser, ConfigSetsAPITest, TestLuceneMatchVersion, TestFieldCache, DistributedQueryComponentCustomSortTest, RollingRestartTest, TestDownShardTolerantSearch, AtomicUpdateProcessorFactoryTest, HdfsCollectionsAPIDistributedZkTest, DistributedQueryElevationComponentTest, TestPHPSerializedResponseWriter, DocValuesNotIndexedTest, MetricsHandlerTest, CleanupOldIndexTest, TestRandomRequestDistribution, TestGenericDistributedQueue, HdfsBasicDistributedZk2Test, PreAnalyzedUpdateProcessorTest, TestReloadAndDeleteDocs, TestManagedResource, SegmentsInfoRequestHandlerTest, CollectionReloadTest, SolrIndexSplitterTest, TestSearcherReuse, SuggestComponentTest, TestCollationField, HttpPartitionTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HttpPartitionTest -Dtests.seed=D93B52828EC50BB8 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=mt -Dtests.timezone=US/Pacific-New -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | HttpPartitionTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > at __randomizedtesting.SeedInfo.seed([D93B52828EC50BB8]:0)
   [junit4] Completed [720/801 (2!)] on J0 in 7283.71s, 1 test, 2 errors <<< FAILURES!

[...truncated 38279 lines...]
-ecj-javadoc-lint-tests:
    [mkdir] Created dir: /tmp/ecj331236023
 [ecj-lint] Compiling 20 source files to /tmp/ecj331236023
 [ecj-lint] ----------
 [ecj-lint] 1. ERROR in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/RandomGeoPolygonTest.java (at line 27)
 [ecj-lint] import static com.carrotsearch.randomizedtesting.RandomizedTest.randomDouble;
 [ecj-lint]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import com.carrotsearch.randomizedtesting.RandomizedTest.randomDouble is never used
 [ecj-lint] ----------
 [ecj-lint] 1 problem (1 error)

BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/build.xml:642: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/build.xml:101: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/build.xml:208: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/common-build.xml:2264: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/common-build.xml:2095: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/common-build.xml:2128: Compile failed; see the compiler error output for details.

Total time: 222 minutes 37 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-Tests-7.x - Build # 40 - Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/40/

6 tests failed.
FAILED:  org.apache.solr.cloud.LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud

Error Message:
Could not find collection : legacyFalse

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : legacyFalse
        at __randomizedtesting.SeedInfo.seed([CE06C185C04F8F23:1F01330064400411]:0)
        at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:118)
        at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:256)
        at org.apache.solr.cloud.LegacyCloudClusterPropTest.checkMandatoryProps(LegacyCloudClusterPropTest.java:154)
        at org.apache.solr.cloud.LegacyCloudClusterPropTest.createAndTest(LegacyCloudClusterPropTest.java:91)
        at org.apache.solr.cloud.LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud(LegacyCloudClusterPropTest.java:71)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestPullReplica.testKillLeader

Error Message:
Replica core_node4 not up to date after 10 seconds expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: Replica core_node4 not up to date after 10 seconds expected:<1> but was:<0>
        at __randomizedtesting.SeedInfo.seed([CE06C185C04F8F23:87103531A2F41B75]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.TestPullReplica.waitForNumDocsInAllReplicas(TestPullReplica.java:541)
        at org.apache.solr.cloud.TestPullReplica.doTestNoLeader(TestPullReplica.java:489)
        at org.apache.solr.cloud.TestPullReplica.testKillLeader(TestPullReplica.java:308)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 8 in http://127.0.0.1:39694/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 8 in http://127.0.0.1:39694/solr
        at __randomizedtesting.SeedInfo.seed([CE06C185C04F8F23:FF6B829ED1F4584]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:889)
        at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:603)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCollectionsAPI

Error Message:
Error from server at https://127.0.0.1:36918/solr/awhollynewcollection_0: No registered leader was found after waiting for 4000ms , collection: awhollynewcollection_0 slice: shard3 saw state=DocCollection(awhollynewcollection_0//collections/awhollynewcollection_0/state.json/9)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-bfffffff",       "state":"active",       "replicas":{         "core_node2":{           "core":"awhollynewcollection_0_shard1_replica_n1",           "base_url":"https://127.0.0.1:40917/solr",           "node_name":"127.0.0.1:40917_solr",           "state":"down",           "type":"NRT"},         "core_node4":{           "core":"awhollynewcollection_0_shard1_replica_n3",           "base_url":"https://127.0.0.1:44279/solr",           "node_name":"127.0.0.1:44279_solr",           "state":"active",           "type":"NRT",           "leader":"true"}}},     "shard2":{       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{         "core_node6":{           "core":"awhollynewcollection_0_shard2_replica_n5",           "base_url":"https://127.0.0.1:44116/solr",           "node_name":"127.0.0.1:44116_solr",           "state":"active",           "type":"NRT"},         "core_node8":{           "core":"awhollynewcollection_0_shard2_replica_n7",           "base_url":"https://127.0.0.1:36918/solr",           "node_name":"127.0.0.1:36918_solr",           "state":"active",           "type":"NRT",           "leader":"true"}}},     "shard3":{       "range":"0-3fffffff",       "state":"active",       "replicas":{         "core_node10":{           "core":"awhollynewcollection_0_shard3_replica_n9",           "base_url":"https://127.0.0.1:40917/solr",           "node_name":"127.0.0.1:40917_solr",           "state":"down",           "type":"NRT"},         "core_node12":{           "core":"awhollynewcollection_0_shard3_replica_n11",           "base_url":"https://127.0.0.1:44279/solr",           "node_name":"127.0.0.1:44279_solr",           "state":"active",           "type":"NRT"}}},     "shard4":{       "range":"40000000-7fffffff",       "state":"active",       "replicas":{         "core_node14":{           "core":"awhollynewcollection_0_shard4_replica_n13",           "base_url":"https://127.0.0.1:44116/solr",           "node_name":"127.0.0.1:44116_solr",           "state":"active",           "type":"NRT"},         "core_node16":{           "core":"awhollynewcollection_0_shard4_replica_n15",           "base_url":"https://127.0.0.1:36918/solr",           "node_name":"127.0.0.1:36918_solr",           "state":"active",           "type":"NRT",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"3",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0"} with live_nodes=[127.0.0.1:44116_solr, 127.0.0.1:36918_solr, 127.0.0.1:40917_solr, 127.0.0.1:44279_solr]

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:36918/solr/awhollynewcollection_0: No registered leader was found after waiting for 4000ms , collection: awhollynewcollection_0 slice: shard3 saw state=DocCollection(awhollynewcollection_0//collections/awhollynewcollection_0/state.json/9)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{
        "core_node2":{
          "core":"awhollynewcollection_0_shard1_replica_n1",
          "base_url":"https://127.0.0.1:40917/solr",
          "node_name":"127.0.0.1:40917_solr",
          "state":"down",
          "type":"NRT"},
        "core_node4":{
          "core":"awhollynewcollection_0_shard1_replica_n3",
          "base_url":"https://127.0.0.1:44279/solr",
          "node_name":"127.0.0.1:44279_solr",
          "state":"active",
          "type":"NRT",
          "leader":"true"}}},
    "shard2":{
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node6":{
          "core":"awhollynewcollection_0_shard2_replica_n5",
          "base_url":"https://127.0.0.1:44116/solr",
          "node_name":"127.0.0.1:44116_solr",
          "state":"active",
          "type":"NRT"},
        "core_node8":{
          "core":"awhollynewcollection_0_shard2_replica_n7",
          "base_url":"https://127.0.0.1:36918/solr",
          "node_name":"127.0.0.1:36918_solr",
          "state":"active",
          "type":"NRT",
          "leader":"true"}}},
    "shard3":{
      "range":"0-3fffffff",
      "state":"active",
      "replicas":{
        "core_node10":{
          "core":"awhollynewcollection_0_shard3_replica_n9",
          "base_url":"https://127.0.0.1:40917/solr",
          "node_name":"127.0.0.1:40917_solr",
          "state":"down",
          "type":"NRT"},
        "core_node12":{
          "core":"awhollynewcollection_0_shard3_replica_n11",
          "base_url":"https://127.0.0.1:44279/solr",
          "node_name":"127.0.0.1:44279_solr",
          "state":"active",
          "type":"NRT"}}},
    "shard4":{
      "range":"40000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node14":{
          "core":"awhollynewcollection_0_shard4_replica_n13",
          "base_url":"https://127.0.0.1:44116/solr",
          "node_name":"127.0.0.1:44116_solr",
          "state":"active",
          "type":"NRT"},
        "core_node16":{
          "core":"awhollynewcollection_0_shard4_replica_n15",
          "base_url":"https://127.0.0.1:36918/solr",
          "node_name":"127.0.0.1:36918_solr",
          "state":"active",
          "type":"NRT",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"3",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0"} with live_nodes=[127.0.0.1:44116_solr, 127.0.0.1:36918_solr, 127.0.0.1:40917_solr, 127.0.0.1:44279_solr]
        at __randomizedtesting.SeedInfo.seed([CE06C185C04F8F23:8673B531C67CA0B6]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1106)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
        at org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:464)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.api.collections.TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete

Error Message:
Error from server at https://127.0.0.1:46373/solr/testcollection_shard1_replica_n2: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/testcollection_shard1_replica_n2/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/testcollection_shard1_replica_n2/update. Reason: <pre>    Can not find: /solr/testcollection_shard1_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.8.v20171121</a><hr/>  </body> </html>

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://127.0.0.1:46373/solr/testcollection_shard1_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/testcollection_shard1_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n2/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.8.v20171121</a><hr/>

</body>
</html>

        at __randomizedtesting.SeedInfo.seed([CE06C185C04F8F23:6DFC6F2047A76586]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1015)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
        at org.apache.solr.cloud.api.collections.TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete(TestCollectionsAPIViaSolrCloudCluster.java:170)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:46373/solr/testcollection_shard1_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/testcollection_shard1_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n2/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.8.v20171121</a><hr/>

</body>
</html>

        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:192)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testBasic

Error Message:
java.util.concurrent.ExecutionException: java.io.IOException: org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{suspend-trigger={name=.scheduled_maintenance}, errorMessages=[No trigger exists with name: .scheduled_maintenance]}],

Stack Trace:
java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{suspend-trigger={name=.scheduled_maintenance}, errorMessages=[No trigger exists with name: .scheduled_maintenance]}],
        at __randomizedtesting.SeedInfo.seed([CE06C185C04F8F23:65FCDC901F93090D]:0)
        at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:540)
        at org.apache.solr.cloud.autoscaling.sim.SimCloudManager$1.request(SimCloudManager.java:414)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.setupTest(TestLargeCluster.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:968)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{suspend-trigger={name=.scheduled_maintenance}, errorMessages=[No trigger exists with name: .scheduled_maintenance]}],
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:538)
        ... 41 more
Caused by: java.io.IOException: org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{suspend-trigger={name=.scheduled_maintenance}, errorMessages=[No trigger exists with name: .scheduled_maintenance]}],
        at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:578)
        at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.lambda$request$12(SimCloudManager.java:537)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:192)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more
Caused by: org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{suspend-trigger={name=.scheduled_maintenance}, errorMessages=[No trigger exists with name: .scheduled_maintenance]}],
        at org.apache.solr.cloud.autoscaling.AutoScalingHandler.processOps(AutoScalingHandler.java:210)
        at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleRequestBody(AutoScalingHandler.java:148)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:575)
        ... 6 more




Build Log:
[...truncated 12811 lines...]
   [junit4] Suite: org.apache.solr.cloud.api.collections.TestCollectionsAPIViaSolrCloudCluster
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.api.collections.TestCollectionsAPIViaSolrCloudCluster_CE06C185C04F8F23-001/init-core-data-001
   [junit4]   2> 416235 WARN  (SUITE-TestCollectionsAPIViaSolrCloudCluster-seed#[CE06C185C04F8F23]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=5 numCloses=5
   [junit4]   2> 416235 INFO  (SUITE-TestCollectionsAPIViaSolrCloudCluster-seed#[CE06C185C04F8F23]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 416237 INFO  (SUITE-TestCollectionsAPIViaSolrCloudCluster-seed#[CE06C185C04F8F23]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 416237 INFO  (SUITE-TestCollectionsAPIViaSolrCloudCluster-seed#[CE06C185C04F8F23]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 416240 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 5 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.api.collections.TestCollectionsAPIViaSolrCloudCluster_CE06C185C04F8F23-001/tempDir-001
   [junit4]   2> 416240 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 416240 INFO  (Thread-1193) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 416240 INFO  (Thread-1193) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 416242 ERROR (Thread-1193) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 416341 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.c.ZkTestServer start zk server on port:45285
   [junit4]   2> 416346 INFO  (zkConnectionManagerCallback-982-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416378 INFO  (jetty-launcher-979-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 416378 INFO  (jetty-launcher-979-thread-3) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 416380 INFO  (jetty-launcher-979-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 416380 INFO  (jetty-launcher-979-thread-4) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 416382 INFO  (jetty-launcher-979-thread-5) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 416383 INFO  (jetty-launcher-979-thread-5) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 416383 INFO  (jetty-launcher-979-thread-5) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 416384 INFO  (jetty-launcher-979-thread-5) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 416384 INFO  (jetty-launcher-979-thread-5) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@23ce6e24{/solr,null,AVAILABLE}
   [junit4]   2> 416385 INFO  (jetty-launcher-979-thread-5) [    ] o.e.j.s.AbstractConnector Started ServerConnector@9b270ca{SSL,[ssl, http/1.1]}{127.0.0.1:43194}
   [junit4]   2> 416385 INFO  (jetty-launcher-979-thread-5) [    ] o.e.j.s.Server Started @416422ms
   [junit4]   2> 416385 INFO  (jetty-launcher-979-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43194}
   [junit4]   2> 416385 ERROR (jetty-launcher-979-thread-5) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 416385 INFO  (jetty-launcher-979-thread-5) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 416385 INFO  (jetty-launcher-979-thread-5) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.4.0
   [junit4]   2> 416385 INFO  (jetty-launcher-979-thread-5) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 416385 INFO  (jetty-launcher-979-thread-5) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 416385 INFO  (jetty-launcher-979-thread-5) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-04-15T12:19:07.354Z
   [junit4]   2> 416387 INFO  (jetty-launcher-979-thread-4) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 416387 INFO  (jetty-launcher-979-thread-4) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 416387 INFO  (jetty-launcher-979-thread-4) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 416387 INFO  (jetty-launcher-979-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7eabff23{/solr,null,AVAILABLE}
   [junit4]   2> 416388 INFO  (jetty-launcher-979-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1ba4845d{SSL,[ssl, http/1.1]}{127.0.0.1:37940}
   [junit4]   2> 416388 INFO  (jetty-launcher-979-thread-4) [    ] o.e.j.s.Server Started @416425ms
   [junit4]   2> 416388 INFO  (jetty-launcher-979-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37940}
   [junit4]   2> 416388 ERROR (jetty-launcher-979-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 416388 INFO  (jetty-launcher-979-thread-4) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 416388 INFO  (jetty-launcher-979-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.4.0
   [junit4]   2> 416388 INFO  (jetty-launcher-979-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 416388 INFO  (jetty-launcher-979-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 416388 INFO  (jetty-launcher-979-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-04-15T12:19:07.357Z
   [junit4]   2> 416394 INFO  (jetty-launcher-979-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 416394 INFO  (jetty-launcher-979-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 416394 INFO  (jetty-launcher-979-thread-3) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 416394 INFO  (jetty-launcher-979-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 416394 INFO  (jetty-launcher-979-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 416394 INFO  (jetty-launcher-979-thread-2) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 416394 INFO  (zkConnectionManagerCallback-986-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416394 INFO  (jetty-launcher-979-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@58a23a0c{/solr,null,AVAILABLE}
   [junit4]   2> 416395 INFO  (jetty-launcher-979-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 416398 INFO  (jetty-launcher-979-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 416398 INFO  (jetty-launcher-979-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 416398 INFO  (jetty-launcher-979-thread-1) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 416398 INFO  (jetty-launcher-979-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5a3f4a2c{/solr,null,AVAILABLE}
   [junit4]   2> 416399 INFO  (jetty-launcher-979-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7ce98e46{SSL,[ssl, http/1.1]}{127.0.0.1:44816}
   [junit4]   2> 416399 INFO  (jetty-launcher-979-thread-1) [    ] o.e.j.s.Server Started @416436ms
   [junit4]   2> 416399 INFO  (jetty-launcher-979-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44816}
   [junit4]   2> 416399 ERROR (jetty-launcher-979-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 416399 INFO  (jetty-launcher-979-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 416399 INFO  (jetty-launcher-979-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.4.0
   [junit4]   2> 416399 INFO  (jetty-launcher-979-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 416399 INFO  (jetty-launcher-979-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 416399 INFO  (jetty-launcher-979-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-04-15T12:19:07.368Z
   [junit4]   2> 416402 INFO  (jetty-launcher-979-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@685bd767{/solr,null,AVAILABLE}
   [junit4]   2> 416402 INFO  (jetty-launcher-979-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5517f5d1{SSL,[ssl, http/1.1]}{127.0.0.1:35413}
   [junit4]   2> 416402 INFO  (jetty-launcher-979-thread-3) [    ] o.e.j.s.Server Started @416440ms
   [junit4]   2> 416402 INFO  (zkConnectionManagerCallback-984-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416402 INFO  (jetty-launcher-979-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35413}
   [junit4]   2> 416403 ERROR (jetty-launcher-979-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 416403 INFO  (jetty-launcher-979-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 416403 INFO  (jetty-launcher-979-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.4.0
   [junit4]   2> 416403 INFO  (jetty-launcher-979-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 416403 INFO  (jetty-launcher-979-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 416403 INFO  (jetty-launcher-979-thread-5) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 416403 INFO  (jetty-launcher-979-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-04-15T12:19:07.372Z
   [junit4]   2> 416409 INFO  (jetty-launcher-979-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@67b6b402{SSL,[ssl, http/1.1]}{127.0.0.1:37627}
   [junit4]   2> 416409 INFO  (jetty-launcher-979-thread-2) [    ] o.e.j.s.Server Started @416447ms
   [junit4]   2> 416409 INFO  (jetty-launcher-979-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37627}
   [junit4]   2> 416409 ERROR (jetty-launcher-979-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 416409 INFO  (jetty-launcher-979-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 416409 INFO  (jetty-launcher-979-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.4.0
   [junit4]   2> 416409 INFO  (jetty-launcher-979-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 416409 INFO  (jetty-launcher-979-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 416410 INFO  (jetty-launcher-979-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-04-15T12:19:07.378Z
   [junit4]   2> 416411 INFO  (zkConnectionManagerCallback-992-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416411 INFO  (zkConnectionManagerCallback-988-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416412 INFO  (jetty-launcher-979-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 416413 INFO  (jetty-launcher-979-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 416414 INFO  (jetty-launcher-979-thread-5) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45285/solr
   [junit4]   2> 416418 INFO  (zkConnectionManagerCallback-990-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416428 INFO  (jetty-launcher-979-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45285/solr
   [junit4]   2> 416428 INFO  (zkConnectionManagerCallback-996-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416430 INFO  (jetty-launcher-979-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 416430 INFO  (zkConnectionManagerCallback-1000-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416431 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1016d8127b30006, likely client has closed socket
   [junit4]   2> 416432 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1016d8127b30007, likely client has closed socket
   [junit4]   2> 416434 INFO  (jetty-launcher-979-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45285/solr
   [junit4]   2> 416439 INFO  (zkConnectionManagerCallback-1006-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416450 INFO  (zkConnectionManagerCallback-1004-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416453 INFO  (jetty-launcher-979-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45285/solr
   [junit4]   2> 416454 INFO  (jetty-launcher-979-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45285/solr
   [junit4]   2> 416467 INFO  (zkConnectionManagerCallback-1008-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416468 INFO  (zkConnectionManagerCallback-1016-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416470 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1016d8127b3000b, likely client has closed socket
   [junit4]   2> 416470 INFO  (zkConnectionManagerCallback-1020-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416471 INFO  (zkConnectionManagerCallback-1013-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416473 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1016d8127b3000d, likely client has closed socket
   [junit4]   2> 416483 INFO  (jetty-launcher-979-thread-4) [n:127.0.0.1:37940_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 416484 INFO  (jetty-launcher-979-thread-1) [n:127.0.0.1:44816_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 416485 INFO  (jetty-launcher-979-thread-1) [n:127.0.0.1:44816_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:44816_solr
   [junit4]   2> 416486 INFO  (jetty-launcher-979-thread-1) [n:127.0.0.1:44816_solr    ] o.a.s.c.Overseer Overseer (id=72459470498889740-127.0.0.1:44816_solr-n_0000000000) starting
   [junit4]   2> 416495 INFO  (zkConnectionManagerCallback-1022-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416496 INFO  (jetty-launcher-979-thread-4) [n:127.0.0.1:37940_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37940_solr
   [junit4]   2> 416509 INFO  (zkCallback-1007-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 416509 INFO  (zkConnectionManagerCallback-1035-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416510 INFO  (jetty-launcher-979-thread-1) [n:127.0.0.1:44816_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 416512 INFO  (jetty-launcher-979-thread-1) [n:127.0.0.1:44816_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45285/solr ready
   [junit4]   2> 416512 INFO  (jetty-launcher-979-thread-3) [n:127.0.0.1:35413_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 416514 INFO  (jetty-launcher-979-thread-3) [n:127.0.0.1:35413_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 416516 INFO  (jetty-launcher-979-thread-3) [n:127.0.0.1:35413_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 416516 INFO  (jetty-launcher-979-thread-3) [n:127.0.0.1:35413_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35413_solr
   [junit4]   2> 416517 INFO  (zkConnectionManagerCallback-1028-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416518 INFO  (zkCallback-1007-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 416534 INFO  (jetty-launcher-979-thread-2) [n:127.0.0.1:37627_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 416541 INFO  (zkCallback-1019-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 416541 INFO  (zkCallback-1034-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 416551 INFO  (jetty-launcher-979-thread-2) [n:127.0.0.1:37627_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 416552 INFO  (jetty-launcher-979-thread-1) [n:127.0.0.1:44816_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 416552 INFO  (jetty-launcher-979-thread-1) [n:127.0.0.1:44816_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44816_solr
   [junit4]   2> 416566 INFO  (zkCallback-1021-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 416570 INFO  (zkCallback-1019-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 416570 INFO  (zkCallback-1007-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 416571 INFO  (zkCallback-1021-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 416571 INFO  (zkCallback-1034-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 416574 INFO  (jetty-launcher-979-thread-2) [n:127.0.0.1:37627_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 416574 INFO  (jetty-launcher-979-thread-2) [n:127.0.0.1:37627_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37627_solr
   [junit4]   2> 416578 INFO  (zkCallback-1021-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 416579 INFO  (zkCallback-1034-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 416582 INFO  (zkCallback-1007-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 416589 INFO  (zkCallback-1027-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 416602 INFO  (zkCallback-1019-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 416602 INFO  (zkCallback-1027-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 416612 INFO  (jetty-launcher-979-thread-5) [n:127.0.0.1:43194_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 416614 INFO  (jetty-launcher-979-thread-5) [n:127.0.0.1:43194_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 416616 INFO  (jetty-launcher-979-thread-5) [n:127.0.0.1:43194_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 416616 INFO  (jetty-launcher-979-thread-5) [n:127.0.0.1:43194_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43194_solr
   [junit4]   2> 416617 INFO  (zkCallback-1021-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 416617 INFO  (zkCallback-1007-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 416617 INFO  (zkCallback-1034-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 416618 INFO  (zkCallback-1019-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 416652 INFO  (zkCallback-1027-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 416675 INFO  (zkCallback-1005-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 416759 INFO  (jetty-launcher-979-thread-3) [n:127.0.0.1:35413_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35413.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 416766 INFO  (jetty-launcher-979-thread-3) [n:127.0.0.1:35413_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35413.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 416766 INFO  (jetty-launcher-979-thread-3) [n:127.0.0.1:35413_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35413.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 416767 INFO  (jetty-launcher-979-thread-3) [n:127.0.0.1:35413_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.api.collections.TestCollectionsAPIViaSolrCloudCluster_CE06C185C04F8F23-001/tempDir-001/node3/.
   [junit4]   2> 416773 INFO  (zkConnectionManagerCallback-1042-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416774 INFO  (jetty-launcher-979-thread-3) [n:127.0.0.1:35413_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 416774 INFO  (jetty-launcher-979-thread-3) [n:127.0.0.1:35413_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45285/solr ready
   [junit4]   2> 416793 INFO  (jetty-launcher-979-thread-1) [n:127.0.0.1:44816_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44816.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 416803 INFO  (jetty-launcher-979-thread-1) [n:127.0.0.1:44816_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44816.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 416803 INFO  (jetty-launcher-979-thread-1) [n:127.0.0.1:44816_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44816.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 416804 INFO  (jetty-launcher-979-thread-1) [n:127.0.0.1:44816_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.api.collections.TestCollectionsAPIViaSolrCloudCluster_CE06C185C04F8F23-001/tempDir-001/node1/.
   [junit4]   2> 416861 INFO  (jetty-launcher-979-thread-2) [n:127.0.0.1:37627_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37627.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 416884 INFO  (jetty-launcher-979-thread-2) [n:127.0.0.1:37627_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37627.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 416885 INFO  (jetty-launcher-979-thread-2) [n:127.0.0.1:37627_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37627.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 416886 INFO  (jetty-launcher-979-thread-2) [n:127.0.0.1:37627_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.api.collections.TestCollectionsAPIViaSolrCloudCluster_CE06C185C04F8F23-001/tempDir-001/node2/.
   [junit4]   2> 416924 INFO  (zkConnectionManagerCallback-1047-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416925 INFO  (jetty-launcher-979-thread-2) [n:127.0.0.1:37627_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 416926 INFO  (jetty-launcher-979-thread-2) [n:127.0.0.1:37627_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45285/solr ready
   [junit4]   2> 416963 INFO  (jetty-launcher-979-thread-5) [n:127.0.0.1:43194_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43194.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 416970 INFO  (jetty-launcher-979-thread-5) [n:127.0.0.1:43194_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43194.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 416970 INFO  (jetty-launcher-979-thread-5) [n:127.0.0.1:43194_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43194.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 416974 INFO  (jetty-launcher-979-thread-5) [n:127.0.0.1:43194_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.api.collections.TestCollectionsAPIViaSolrCloudCluster_CE06C185C04F8F23-001/tempDir-001/node5/.
   [junit4]   2> 416983 INFO  (jetty-launcher-979-thread-4) [n:127.0.0.1:37940_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37940.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 416991 INFO  (jetty-launcher-979-thread-4) [n:127.0.0.1:37940_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37940.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 416991 INFO  (jetty-launcher-979-thread-4) [n:127.0.0.1:37940_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37940.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 416992 INFO  (jetty-launcher-979-thread-4) [n:127.0.0.1:37940_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.api.collections.TestCollectionsAPIViaSolrCloudCluster_CE06C185C04F8F23-001/tempDir-001/node4/.
   [junit4]   2> 417009 INFO  (zkConnectionManagerCallback-1052-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 417010 INFO  (jetty-launcher-979-thread-5) [n:127.0.0.1:43194_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 417011 INFO  (jetty-launcher-979-thread-5) [n:127.0.0.1:43194_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45285/solr ready
   [junit4]   2> 417037 INFO  (zkConnectionManagerCallback-1057-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 417038 INFO  (jetty-launcher-979-thread-4) [n:127.0.0.1:37940_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 417039 INFO  (jetty-launcher-979-thread-4) [n:127.0.0.1:37940_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45285/solr ready
   [junit4]   2> 417084 INFO  (zkConnectionManagerCallback-1059-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 417090 INFO  (zkConnectionManagerCallback-1064-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 417090 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 417091 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45285/solr ready
   [junit4]   2> 417094 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCollectionCreateSearchDelete
   [junit4]   2> 417096 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@5517f5d1{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 417096 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=622351640
   [junit4]   2> 417096 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 417096 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4322ef4c: rootName = solr_35413, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@5f610a31
   [junit4]   2> 417102 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 417102 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3286d1ce: rootName = solr_35413, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@540c433d
   [junit4]   2> 417105 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 417105 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@51503353: rootName = solr_35413, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@51109815
   [junit4]   2> 417110 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:35413_solr
   [junit4]   2> 417111 INFO  (zkCallback-1027-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
   [junit4]   2> 417112 INFO  (zkCallback-1007-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
   [junit4]   2> 417116 INFO  (zkCallback-1019-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
   [junit4]   2> 417116 INFO  (zkCallback-1034-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
   [junit4]   2> 417116 INFO  (zkCallback-1021-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
   [junit4]   2> 417116 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 417131 INFO  (zkCallback-1005-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
   [junit4]   2> 417133 INFO  (zkCallback-1051-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
   [junit4]   2> 417133 INFO  (zkCallback-1063-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
   [junit4]   2> 417133 INFO  (zkCallback-1041-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
   [junit4]   2> 417133 INFO  (zkCallback-1056-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
   [junit4]   2> 417133 INFO  (zkCallback-1046-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
   [junit4]   2> 417139 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@58a23a0c{/solr,null,UNAVAILABLE}
   [junit4]   2> 417140 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 417146 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 417214 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 417214 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 417214 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 417218 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3dcc7d20{/solr,null,AVAILABLE}
   [junit4]   2> 417218 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@d9af01e{SSL,[ssl, http/1.1]}{127.0.0.1:46373}
   [junit4]   2> 417219 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.e.j.s.Server Started @417256ms
   [junit4]   2> 417219 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46373}
   [junit4]   2> 417219 ERROR (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 417226 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 417226 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.4.0
   [junit4]   2> 417226 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 417226 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 417226 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-04-15T12:19:08.195Z
   [junit4]   2> 417232 INFO  (zkConnectionManagerCallback-1066-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 417233 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 417243 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45285/solr
   [junit4]   2> 417246 INFO  (zkConnectionManagerCallback-1070-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 417253 INFO  (zkConnectionManagerCallback-1072-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 417261 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [n:127.0.0.1:46373_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 417268 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [n:127.0.0.1:46373_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 417270 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [n:127.0.0.1:46373_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 417270 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [n:127.0.0.1:46373_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46373_solr
   [junit4]   2> 417271 INFO  (zkCallback-1051-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 417271 INFO  (zkCallback-1063-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 417271 INFO  (zkCallback-1027-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 417271 INFO  (zkCallback-1007-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 417271 INFO  (zkCallback-1005-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 417271 INFO  (zkCallback-1046-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 417274 INFO  (zkCallback-1034-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 417274 INFO  (zkCallback-1019-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 417274 INFO  (zkCallback-1056-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 417290 INFO  (zkCallback-1071-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 417433 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [n:127.0.0.1:46373_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46373.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 417441 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [n:127.0.0.1:46373_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46373.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 417441 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [n:127.0.0.1:46373_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46373.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c1754fa
   [junit4]   2> 417442 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [n:127.0.0.1:46373_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.api.collections.TestCollectionsAPIViaSolrCloudCluster_CE06C185C04F8F23-001/tempDir-001/node6/.
   [junit4]   2> 417454 INFO  (zkConnectionManagerCallback-1079-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 417455 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [n:127.0.0.1:46373_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 417455 INFO  (TEST-TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete-seed#[CE06C185C04F8F23]) [n:127.0.0.1:46373_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45285/solr ready
   [junit4]   2> 417544 INFO  (qtp1663875874-4066) [n:127.0.0.1:37627_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=solrCloudCollectionConfig&maxShardsPerNode=1&name=testcollection&nrtReplicas=2&action=CREATE&numShards=2&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 417546 INFO  (OverseerThreadFactory-1770-thread-1) [    ] o.a.s.c.a.c.CreateCollectionCmd Create collection testcollection
   [junit4]   2> 417654 INFO  (OverseerStateUpdate-72459470498889740-127.0.0.1:44816_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testcollection_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:43194/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 417657 INFO  (OverseerStateUpdate-72459470498889740-127.0.0.1:44816_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testcollection_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:46373/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 417659 INFO  (OverseerStateUpdate-72459470498889740-127.0.0.1:44816_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testcollection_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:44816/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 417661 INFO  (OverseerStateUpdate-72459470498889740-127.0.0.1:44816_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testcollection_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:37627/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 417909 INFO  (qtp1663875874-4099) [n:127.0.0.1:37627_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&coreNodeName=core_node8&name=testcollection_shard2_replica_n6&action=CREATE&numShards=2&shard=shard2&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin
   [junit4]   2> 417932 INFO  (qtp696035590-4226) [n:127.0.0.1:46373_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&coreNodeName=core_node5&name=testcollection_shard1_replica_n2&action=CREATE&numShards=2&shard=shard1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin
   [junit4]   2> 417933 INFO  (qtp401764514-4108) [n:127.0.0.1:44816_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&coreNodeName=core_node7&name=testcollection_shard2_replica_n4&action=CREATE&numShards=2&shard=shard2&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin
   [junit4]   2> 417939 INFO  (qtp1475404261-4077) [n:127.0.0.1:43194_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&coreNodeName=core_node3&name=testcollection_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin
   [junit4]   2> 418063 INFO  (zkCallback-1071-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testcollection/state.json] for collection [testcollection] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 418063 INFO  (zkCallback-1019-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testcollection/state.json] for collection [testcollection] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 418063 INFO  (zkCallback-1027-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testcollection/state.json] for collection [testcollection] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 418063 INFO  (zkCallback-1027-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testcollection/state.json] for collection [testcollection] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 418063 INFO  (zkCallback-1005-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testcollection/state.json] for collection [testcollection] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 418066 INFO  (zkCallback-1019-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testcollection/state.json] for collection [testcollection] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 418075 INFO  (zkCallback-1071-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path

[...truncated too long message...]

ure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null111606620
     [copy] Copying 40 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null111606620
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null111606620

resolve-example:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve-server:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1110868827
     [copy] Copying 246 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1110868827
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1110868827

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 52ms :: artifacts dl 1ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 128 minutes 14 seconds
Archiving artifacts
WARN: No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
WARN: java.lang.InterruptedException: no matches found within 10000
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-Tests-7.x - Build # 41 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/41/

7 tests failed.
FAILED:  org.apache.solr.cloud.TestRebalanceLeaders.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:36878

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:36878
        at __randomizedtesting.SeedInfo.seed([4C785A114CCDD246:C42C65CBE231BFBE]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1106)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:425)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:341)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 44 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestRebalanceLeaders

Error Message:
98 threads leaked from SUITE scope at org.apache.solr.cloud.TestRebalanceLeaders:     1) Thread[id=11148, name=qtp1110176047-11148, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=11319, name=ScheduledTrigger-2864-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=11215, name=qtp2019149649-11215, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=11149, name=qtp1110176047-11149, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=11168, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=11212, name=searcherExecutor-2852-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=11158, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    8) Thread[id=11112, name=qtp1213989072-11112, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)    9) Thread[id=11236, name=Thread-2409, state=WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1737)   10) Thread[id=11334, name=searcherExecutor-2872-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   11) Thread[id=11306, name=zkConnectionManagerCallback-3323-thread-1-EventThread, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)   12) Thread[id=11164, name=zkConnectionManagerCallback-3308-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   13) Thread[id=11301, name=zkConnectionManagerCallback-3338-thread-1-SendThread(127.0.0.1:46496), state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)   14) Thread[id=11128, name=zkConnectionManagerCallback-3293-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   15) Thread[id=11325, name=ScheduledTrigger-2864-thread-2, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   16) Thread[id=11227, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   17) Thread[id=11278, name=zkConnectionManagerCallback-3330-thread-1-EventThread, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)   18) Thread[id=11142, name=searcherExecutor-2830-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   19) Thread[id=11153, name=Scheduler-1954250675, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   20) Thread[id=11171, name=zkConnectionManagerCallback-3315-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   21) Thread[id=11218, name=qtp2019149649-11218, state=RUNNABLE, group=TGRP-TestRebalanceLeaders]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   22) Thread[id=11214, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   23) Thread[id=11242, name=Scheduler-399728715, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   24) Thread[id=11309, name=zkConnectionManagerCallback-3308-thread-1-EventThread, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)   25) Thread[id=11187, name=qtp1757433187-11187, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   26) Thread[id=11183, name=qtp1757433187-11183, state=RUNNABLE, group=TGRP-TestRebalanceLeaders]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   27) Thread[id=11185, name=qtp1757433187-11185, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   28) Thread[id=11223, name=Scheduler-1516335860, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   29) Thread[id=11114, name=qtp1213989072-11114, state=RUNNABLE, group=TGRP-TestRebalanceLeaders]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   30) Thread[id=11182, name=qtp1757433187-11182-acceptor-0@6197bf3e-ServerConnector@722b7581{HTTP/1.1,[http/1.1]}{127.0.0.1:39705}, state=RUNNABLE, group=TGRP-TestRebalanceLeaders]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:379)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:638)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   31) Thread[id=11135, name=zkConnectionManagerCallback-3300-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   32) Thread[id=11238, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   33) Thread[id=11228, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   34) Thread[id=11177, name=searcherExecutor-2841-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   35) Thread[id=11147, name=qtp1110176047-11147-acceptor-0@28ab1472-ServerConnector@5b102bee{HTTP/1.1,[http/1.1]}{127.0.0.1:44554}, state=RUNNABLE, group=TGRP-TestRebalanceLeaders]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:379)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:638)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   36) Thread[id=11356, name=AutoscalingActionExecutor-2865-thread-1, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)         at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:316)         at org.apache.solr.client.solrj.impl.ZkDistribStateManager.hasData(ZkDistribStateManager.java:58)         at org.apache.solr.cloud.autoscaling.ScheduledTriggers.waitForPendingTasks(ScheduledTriggers.java:393)         at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:301)         at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$491/964088217.run(Unknown Source)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:192)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$141/864623513.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   37) Thread[id=11188, name=Scheduler-929609296, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   38) Thread[id=11287, name=zkConnectionManagerCallback-3293-thread-1-SendThread(127.0.0.1:46496), state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)   39) Thread[id=11111, name=qtp1213989072-11111-acceptor-0@64bf0eef-ServerConnector@5e9ba06f{HTTP/1.1,[http/1.1]}{127.0.0.1:41350}, state=RUNNABLE, group=TGRP-TestRebalanceLeaders]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:379)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:638)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   40) Thread[id=11216, name=qtp2019149649-11216, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)         at org.apache.solr.cloud.OverseerTaskQueue$LatchWatcher.await(OverseerTaskQueue.java:158)         at org.apache.solr.cloud.OverseerTaskQueue.offer(OverseerTaskQueue.java:208)         at org.apache.solr.handler.admin.CollectionsHandler.sendToOCPQueue(CollectionsHandler.java:347)         at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:257)         at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:231)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)         at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)         at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:393)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:339)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:527)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)         at org.eclipse.jetty.server.Server.handle(Server.java:530)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)         at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)         at org.eclipse.jetty.io.ManagedSelector$$Lambda$223/1158720033.run(Unknown Source)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   41) Thread[id=11299, name=zkConnectionManagerCallback-3308-thread-1-SendThread(127.0.0.1:46496), state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)   42) Thread[id=11117, name=Scheduler-2100102444, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   43) Thread[id=11136, name=Scheduler-909146469, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   44) Thread[id=11145, name=qtp1110176047-11145, state=RUNNABLE, group=TGRP-TestRebalanceLeaders]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)         at org.eclipse.jetty.io.ManagedSelector$$Lambda$223/1158720033.run(Unknown Source)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   45) Thread[id=11302, name=zkConnectionManagerCallback-3338-thread-1-EventThread, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)   46) Thread[id=11144, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   47) Thread[id=11116, name=qtp1213989072-11116, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   48) Thread[id=11132, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   49) Thread[id=11222, name=qtp2019149649-11222, state=RUNNABLE, group=TGRP-TestRebalanceLeaders]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   50) Thread[id=11288, name=zkConnectionManagerCallback-3293-thread-1-EventThread, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)   51) Thread[id=11199, name=zkConnectionManagerCallback-3323-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   52) Thread[id=11150, name=qtp1110176047-11150, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   53) Thread[id=11186, name=qtp1757433187-11186, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   54) Thread[id=11312, name=OverseerAutoScalingTriggerThread-72467330987130911-127.0.0.1:39705_-n_0000000005, state=WAITING, group=Overseer autoscaling triggers]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:181)         at java.lang.Thread.run(Thread.java:748)   55) Thread[id=11180, name=qtp1757433187-11180, state=RUNNABLE, group=TGRP-TestRebalanceLeaders]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   56) Thread[id=11322, name=OverseerThreadFactory-2866-thread-1, state=WAITING, group=Overseer collection creation process.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   57) Thread[id=11121, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   58) Thread[id=11110, name=qtp1213989072-11110, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   59) Thread[id=11206, name=zkConnectionManagerCallback-3330-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   60) Thread[id=11184, name=qtp1757433187-11184, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   61) Thread[id=11327, name=ScheduledTrigger-2864-thread-3, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   62) Thread[id=11220, name=qtp2019149649-11220, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   63) Thread[id=11217, name=qtp2019149649-11217-acceptor-0@14b36c30-ServerConnector@a50f4e0{HTTP/1.1,[http/1.1]}{127.0.0.1:36878}, state=RUNNABLE, group=TGRP-TestRebalanceLeaders]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:379)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:638)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   64) Thread[id=11308, name=zkConnectionManagerCallback-3315-thread-1-EventThread, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)   65) Thread[id=11152, name=qtp1110176047-11152, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   66) Thread[id=11293, name=zkConnectionManagerCallback-3300-thread-1-SendThread(127.0.0.1:46496), state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)   67) Thread[id=11219, name=qtp2019149649-11219, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   68) Thread[id=11113, name=qtp1213989072-11113, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   69) Thread[id=11151, name=qtp1110176047-11151, state=RUNNABLE, group=TGRP-TestRebalanceLeaders]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   70) Thread[id=11311, name=OverseerCollectionConfigSetProcessor-72467330987130911-127.0.0.1:39705_-n_0000000005, state=TIMED_WAITING, group=Overseer collection creation process.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)         at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:392)         at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:193)         at java.lang.Thread.run(Thread.java:748)   71) Thread[id=11203, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   72) Thread[id=11173, name=Scheduler-90989771, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   73) Thread[id=11307, name=zkConnectionManagerCallback-3315-thread-1-SendThread(127.0.0.1:46496), state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)   74) Thread[id=11109, name=qtp1213989072-11109, state=RUNNABLE, group=TGRP-TestRebalanceLeaders]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   75) Thread[id=11115, name=qtp1213989072-11115, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   76) Thread[id=11179, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   77) Thread[id=11241, name=zkConnectionManagerCallback-3345-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   78) Thread[id=11146, name=qtp1110176047-11146, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   79) Thread[id=11305, name=zkConnectionManagerCallback-3323-thread-1-SendThread(127.0.0.1:46496), state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)   80) Thread[id=11234, name=zkConnectionManagerCallback-3338-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   81) Thread[id=11303, name=zkConnectionManagerCallback-3345-thread-1-SendThread(127.0.0.1:46496), state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)   82) Thread[id=11304, name=zkConnectionManagerCallback-3345-thread-1-EventThread, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)   83) Thread[id=11207, name=Scheduler-1698464292, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   84) Thread[id=11277, name=zkConnectionManagerCallback-3330-thread-1-SendThread(127.0.0.1:46496), state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)   85) Thread[id=11221, name=qtp2019149649-11221, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   86) Thread[id=11310, name=OverseerStateUpdate-72467330987130911-127.0.0.1:39705_-n_0000000005, state=TIMED_WAITING, group=Overseer state updater.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)         at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:415)         at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:145)         at java.lang.Thread.run(Thread.java:748)   87) Thread[id=11166, name=Thread-2391, state=WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1737)   88) Thread[id=11130, name=Thread-2382, state=WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1737)   89) Thread[id=11201, name=Thread-2400, state=WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1737)   90) Thread[id=11193, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   91) Thread[id=11157, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   92) Thread[id=11181, name=qtp1757433187-11181, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   93) Thread[id=11329, name=qtp2019149649-11329, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   94) Thread[id=11330, name=ScheduledTrigger-2864-thread-4, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)         at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:316)         at org.apache.solr.client.solrj.impl.ZkDistribStateManager.hasData(ZkDistribStateManager.java:58)         at org.apache.solr.cloud.autoscaling.NodeLostTrigger.removeMarker(NodeLostTrigger.java:182)         at org.apache.solr.cloud.autoscaling.NodeLostTrigger.lambda$run$2(NodeLostTrigger.java:161)         at org.apache.solr.cloud.autoscaling.NodeLostTrigger$$Lambda$492/1587465664.accept(Unknown Source)         at java.util.ArrayList.forEach(ArrayList.java:1257)         at org.apache.solr.cloud.autoscaling.NodeLostTrigger.run(NodeLostTrigger.java:159)         at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.run(ScheduledTriggers.java:589)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   95) Thread[id=11294, name=zkConnectionManagerCallback-3300-thread-1-EventThread, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)   96) Thread[id=11192, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   97) Thread[id=11122, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   98) Thread[id=11332, name=searcherExecutor-2867-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 98 threads leaked from SUITE scope at org.apache.solr.cloud.TestRebalanceLeaders:
   1) Thread[id=11148, name=qtp1110176047-11148, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=11319, name=ScheduledTrigger-2864-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=11215, name=qtp2019149649-11215, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=11149, name=qtp1110176047-11149, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=11168, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=11212, name=searcherExecutor-2852-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=11158, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=11112, name=qtp1213989072-11112, state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)
        at java.lang.Thread.run(Thread.java:748)
   9) Thread[id=11236, name=Thread-2409, state=WAITING, group=TGRP-TestRebalanceLeaders]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.solr.core.CloserThread.run(CoreContainer.java:1737)
  10) Thread[id=11334, name=searcherExecutor-2872-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  11) Thread[id=11306, name=zkConnectionManagerCallback-3323-thread-1-EventThread, state=WAITING, group=TGRP-TestRebalanceLeaders]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
  12) Thread[id=11164, name=zkConnectionManagerCallback-3308-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  13) Thread[id=11301, name=zkConnectionManagerCallback-3338-thread-1-SendThread(127.0.0.1:46496), state=TIMED_WAITING, group=TGRP-TestRebalanceLeaders]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)
  14) Thread[id=11128, name=zkConnectionManagerCallback-3293-thread-1, state=WAITING, group=TGRP-TestRebalanceLeaders]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  15) Thread[id=11325, name=ScheduledTrigger-2864-thread-2, state=WAITING, group=TGRP-TestRebalanceLeaders]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor

[...truncated too long message...]

re] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null705409752
     [copy] Copying 40 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null705409752
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null705409752

resolve-example:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve-server:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1324611724
     [copy] Copying 246 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1324611724
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1324611724

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 49ms :: artifacts dl 21ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 197 minutes 17 seconds
Archiving artifacts
WARN: No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
WARN: java.lang.InterruptedException: no matches found within 10000
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-Tests-7.x - Build # 42 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/42/

All tests passed

Build Log:
[...truncated 62081 lines...]
-ecj-javadoc-lint-tests:
    [mkdir] Created dir: /tmp/ecj1050644116
 [ecj-lint] Compiling 895 source files to /tmp/ecj1050644116
 [ecj-lint] invalid Class-Path header in manifest of jar file: /x1/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/jars/org.restlet-2.3.0.jar
 [ecj-lint] invalid Class-Path header in manifest of jar file: /x1/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.ext.servlet/jars/org.restlet.ext.servlet-2.3.0.jar
 [ecj-lint] ----------
 [ecj-lint] 1. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/core/src/test/org/apache/solr/analysis/TokenizerChainTest.java (at line 37)
 [ecj-lint] TokenizerChain tokenizerChain = new TokenizerChain(
 [ecj-lint]               ^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'tokenizerChain' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 2. ERROR in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/core/src/test/org/apache/solr/cloud/ForceLeaderTest.java (at line 25)
 [ecj-lint] import java.util.concurrent.TimeUnit;
 [ecj-lint]       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import java.util.concurrent.TimeUnit is never used
 [ecj-lint] ----------
 [ecj-lint] 3. ERROR in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/core/src/test/org/apache/solr/cloud/ForceLeaderTest.java (at line 26)
 [ecj-lint] import java.util.stream.Collectors;
 [ecj-lint]       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import java.util.stream.Collectors is never used
 [ecj-lint] ----------
 [ecj-lint] 4. ERROR in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/core/src/test/org/apache/solr/cloud/ForceLeaderTest.java (at line 32)
 [ecj-lint] import org.apache.solr.cloud.overseer.OverseerAction;
 [ecj-lint]       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import org.apache.solr.cloud.overseer.OverseerAction is never used
 [ecj-lint] ----------
 [ecj-lint] 5. ERROR in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/core/src/test/org/apache/solr/cloud/ForceLeaderTest.java (at line 38)
 [ecj-lint] import org.apache.solr.common.cloud.ZkNodeProps;
 [ecj-lint]       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import org.apache.solr.common.cloud.ZkNodeProps is never used
 [ecj-lint] ----------
 [ecj-lint] 6. ERROR in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/core/src/test/org/apache/solr/cloud/ForceLeaderTest.java (at line 39)
 [ecj-lint] import org.apache.solr.common.cloud.ZkStateReader;
 [ecj-lint]       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import org.apache.solr.common.cloud.ZkStateReader is never used
 [ecj-lint] ----------
 [ecj-lint] 7. ERROR in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/core/src/test/org/apache/solr/cloud/ForceLeaderTest.java (at line 41)
 [ecj-lint] import org.apache.solr.common.util.Utils;
 [ecj-lint]       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import org.apache.solr.common.util.Utils is never used
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 8. ERROR in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/core/src/test/org/apache/solr/cloud/MoveReplicaTest.java (at line 25)
 [ecj-lint] import java.util.HashSet;
 [ecj-lint]       ^^^^^^^^^^^^^^^^^
 [ecj-lint] The import java.util.HashSet is never used
 [ecj-lint] ----------
 [ecj-lint] 9. ERROR in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/core/src/test/org/apache/solr/cloud/MoveReplicaTest.java (at line 42)
 [ecj-lint] import org.apache.solr.common.cloud.CollectionStateWatcher;
 [ecj-lint]       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import org.apache.solr.common.cloud.CollectionStateWatcher is never used
 [ecj-lint] ----------
 [ecj-lint] 10. ERROR in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/core/src/test/org/apache/solr/cloud/MoveReplicaTest.java (at line 46)
 [ecj-lint] import org.apache.solr.common.cloud.ZkStateReaderAccessor;
 [ecj-lint]       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import org.apache.solr.common.cloud.ZkStateReaderAccessor is never used
 [ecj-lint] ----------
 [ecj-lint] 10 problems (9 errors, 1 warning)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/build.xml:642: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/build.xml:101: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build.xml:690: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/common-build.xml:2095: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/common-build.xml:2128: Compile failed; see the compiler error output for details.

Total time: 102 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-Tests-7.x - Build # 43 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/43/

2 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
        at __randomizedtesting.SeedInfo.seed([F4FDB65561595C5B:A9B5A8DCAE9FFA14]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate(TestTriggerIntegration.java:1199)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventFromRestoredState

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
        at __randomizedtesting.SeedInfo.seed([F4FDB65561595C5B:F4CB02954854F831]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventFromRestoredState(TestTriggerIntegration.java:691)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12854 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
   [junit4]   2> 592084 INFO  (SUITE-TestTriggerIntegration-seed#[F4FDB65561595C5B]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_F4FDB65561595C5B-001/init-core-data-001
   [junit4]   2> 592086 WARN  (SUITE-TestTriggerIntegration-seed#[F4FDB65561595C5B]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=42 numCloses=42
   [junit4]   2> 592086 INFO  (SUITE-TestTriggerIntegration-seed#[F4FDB65561595C5B]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 592087 INFO  (SUITE-TestTriggerIntegration-seed#[F4FDB65561595C5B]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 592106 INFO  (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTriggerRestoreState
   [junit4]   2> 592110 INFO  (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 592110 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 592110 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 592110 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 592110 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 592110 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 592110 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 592119 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 592120 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 592120 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 592120 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 592134 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10008_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 592146 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 592146 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 592148 DEBUG (ScheduledTrigger-1619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592171 DEBUG (ScheduledTrigger-1619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592191 DEBUG (ScheduledTrigger-1619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592211 DEBUG (ScheduledTrigger-1619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592234 DEBUG (ScheduledTrigger-1619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592247 DEBUG (simCloudManagerPool-1618-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 592247 DEBUG (simCloudManagerPool-1618-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 592256 DEBUG (ScheduledTrigger-1619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592257 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 592257 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 592257 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 592276 DEBUG (ScheduledTrigger-1619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592283 DEBUG (simCloudManagerPool-1618-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 592283 DEBUG (simCloudManagerPool-1618-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 592284 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 592284 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 592284 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger - Initial livenodes: [127.0.0.1:10009_solr, 127.0.0.1:10008_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 592284 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger instantiated with properties: {event=nodeAdded, waitFor=5, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
   [junit4]   2> 592284 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger Adding node from marker path: 127.0.0.1:10009_solr
   [junit4]   2> 592284 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger - removing marker path: /autoscaling/nodeAdded/127.0.0.1:10009_solr
   [junit4]   2> 592285 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 592285 DEBUG (ScheduledTrigger-1619-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 592285 DEBUG (ScheduledTrigger-1619-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 592297 DEBUG (ScheduledTrigger-1619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 592305 DEBUG (ScheduledTrigger-1619-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 592305 DEBUG (ScheduledTrigger-1619-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 592318 DEBUG (ScheduledTrigger-1619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 592325 DEBUG (ScheduledTrigger-1619-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 592325 DEBUG (ScheduledTrigger-1619-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 592338 DEBUG (ScheduledTrigger-1619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 592357 DEBUG (ScheduledTrigger-1619-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 592357 DEBUG (ScheduledTrigger-1619-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 592357 DEBUG (simCloudManagerPool-1618-thread-3) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 592358 DEBUG (simCloudManagerPool-1618-thread-3) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 592358 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 592358 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 592358 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger - Initial livenodes: [127.0.0.1:10009_solr, 127.0.0.1:10008_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 592359 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
   [junit4]   2> 592359 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 592359 DEBUG (ScheduledTrigger-1619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 592359 DEBUG (ScheduledTrigger-1619-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 592359 DEBUG (ScheduledTrigger-1619-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 592359 DEBUG (ScheduledTrigger-1619-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger firing registered processor for nodes: [127.0.0.1:10009_solr] added at times [24361529039765546], now=24361532761714146
   [junit4]   2> 592359 DEBUG (ScheduledTrigger-1619-thread-1) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"568cae7c99c02aTbecttotktkl746gglgm51fi0x",
   [junit4]   2>   "source":"node_added_restore_trigger",
   [junit4]   2>   "eventTime":24361529039765546,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[24361529039765546],
   [junit4]   2>     "nodeNames":["127.0.0.1:10009_solr"]}}
   [junit4]   2> 592359 DEBUG (ScheduledTrigger-1619-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_restore_trigger]
   [junit4]   2> 592414 DEBUG (ScheduledTrigger-1619-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger - removing marker path: /autoscaling/nodeAdded/127.0.0.1:10009_solr
   [junit4]   2> 592415 DEBUG (AutoscalingActionExecutor-1620-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"568cae7c99c02aTbecttotktkl746gglgm51fi0x",
   [junit4]   2>   "source":"node_added_restore_trigger",
   [junit4]   2>   "eventTime":24361529039765546,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[24361529039765546],
   [junit4]   2>     "_enqueue_time_":24361532788546946,
   [junit4]   2>     "nodeNames":["127.0.0.1:10009_solr"]}}
   [junit4]   2> 592416 INFO  (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTriggerRestoreState
   [junit4]   2> 592416 INFO  (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes: 3
   [junit4]   2> ## Empty nodes: 3
   [junit4]   2> ## Dead nodes: 0
   [junit4]   2> ## Collections: []
   [junit4]   2> ## Max replicas per node: 0
   [junit4]   2> ## Min replicas per node: 0
   [junit4]   2> ## Total replicas: 0
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ## - autoscaling        3
   [junit4]   2> ## - update             1
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: node_added_restore_trigger
   [junit4]   2> ## - STARTED         1
   [junit4]   2>
   [junit4]   2> 592420 INFO  (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeMarkersRegistration
   [junit4]   2> 592420 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 592420 INFO  (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 592420 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 592421 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 592421 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 592421 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 592421 WARN  (AutoscalingActionExecutor-1620-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Exception executing actions
   [junit4]   2> org.apache.lucene.store.AlreadyClosedException: ScheduledTrigger node_added_restore_trigger has been closed.
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.dequeue(ScheduledTriggers.java:546) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:319) ~[java/:?]
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_152]
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_152]
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:192) ~[java/:?]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
   [junit4]   2> 592422 DEBUG (AutoscalingActionExecutor-1620-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 374 ms for event id=568cae7c99c02aTbecttotktkl746gglgm51fi0x
   [junit4]   2> 592422 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 592422 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 592422 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 592438 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 592439 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 592439 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion -1
   [junit4]   2> 592439 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 592452 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10009_solr, 127.0.0.1:10008_solr]
   [junit4]   2> 592466 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 592466 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 592467 DEBUG (ScheduledTrigger-1622-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592500 DEBUG (ScheduledTrigger-1622-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592520 DEBUG (ScheduledTrigger-1622-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592550 DEBUG (ScheduledTrigger-1622-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592552 DEBUG (simCloudManagerPool-1621-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 592552 DEBUG (simCloudManagerPool-1621-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 592553 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 592553 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 592553 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 592554 INFO  (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.TestTriggerIntegration ====== KILL OVERSEER 1
   [junit4]   2> 592554 INFO  (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 592554 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 592554 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 592554 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 592554 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 592554 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 592555 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 592555 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 592728 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 592729 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion -1
   [junit4]   2> 592729 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 592729 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10008_solr]
   [junit4]   2> 592729 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:10009_solr
   [junit4]   2> 592738 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 592738 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 592738 DEBUG (ScheduledTrigger-1625-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592738 DEBUG (ScheduledTrigger-1625-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10009_solr
   [junit4]   2> 592759 DEBUG (ScheduledTrigger-1625-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592780 DEBUG (ScheduledTrigger-1625-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592801 DEBUG (ScheduledTrigger-1625-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592821 DEBUG (ScheduledTrigger-1625-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592830 INFO  (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.TestTriggerIntegration ====== ADD TRIGGERS
   [junit4]   2> 592841 DEBUG (ScheduledTrigger-1625-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592859 INFO  (simCloudManagerPool-1624-thread-1) [    ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 592859 DEBUG (simCloudManagerPool-1624-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 592859 DEBUG (simCloudManagerPool-1624-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
   [junit4]   2> 592860 INFO  (simCloudManagerPool-1624-thread-1) [    ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 592860 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 592860 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10008_solr]
   [junit4]   2> 592860 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventMarkerAction}], enabled=true}
   [junit4]   2> 592860 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 592861 DEBUG (ScheduledTrigger-1625-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 592861 DEBUG (ScheduledTrigger-1625-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 592861 DEBUG (ScheduledTrigger-1625-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 592879 INFO  (simCloudManagerPool-1624-thread-2) [    ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 592879 DEBUG (simCloudManagerPool-1624-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 592879 DEBUG (simCloudManagerPool-1624-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
   [junit4]   2> 592880 INFO  (simCloudManagerPool-1624-thread-2) [    ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 592880 INFO  (simCloudManagerPool-1624-thread-2) [    ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 592880 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 592881 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_trigger - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10008_solr]
   [junit4]   2> 592881 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 592881 INFO  (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.TestTriggerIntegration ====== ADD NODE 1
   [junit4]   2> 592881 DEBUG (ScheduledTrigger-1625-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
   [junit4]   2> 592881 DEBUG (ScheduledTrigger-1625-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 592881 DEBUG (ScheduledTrigger-1625-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 592881 DEBUG (ScheduledTrigger-1625-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10011_solr at time 24361558888004696
   [junit4]   2> 592881 DEBUG (ScheduledTrigger-1625-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 592901 DEBUG (ScheduledTrigger-1625-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 3
   [junit4]   2> 592902 DEBUG (ScheduledTrigger-1625-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 592902 DEBUG (ScheduledTrigger-1625-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 592902 DEBUG (ScheduledTrigger-1625-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10011_solr] added at times [24361558888004696], now=24361559904832346
   [junit4]   2> 592902 DEBUG (ScheduledTrigger-1625-thread-3) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"568cb56fb1bc58Tbecttotktkl746gglgm51fi0z",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":24361558888004696,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[24361558888004696],
   [junit4]   2>     "nodeNames":["127.0.0.1:10011_solr"]}}
   [junit4]   2> 592906 DEBUG (ScheduledTrigger-1625-thread-3) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_lost_trigger, node_added_trigger]
   [junit4]   2> 592931 DEBUG (ScheduledTrigger-1625-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - removing marker path: /autoscaling/nodeAdded/127.0.0.1:10011_solr
   [junit4]   2> 592931 DEBUG (AutoscalingActionExecutor-1626-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"568cb56fb1bc58Tbecttotktkl746gglgm51fi0z",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":24361558888004696,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[24361558888004696],
   [junit4]   2>     "_enqueue_time_":24361560126787446,
   [junit4]   2>     "nodeNames":["127.0.0.1:10011_solr"]}}
   [junit4]   2> 592932 DEBUG (AutoscalingActionExecutor-1626-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 592932 DEBUG (AutoscalingActionExecutor-1626-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger after 100ms
   [junit4]   2> 592932 DEBUG (AutoscalingActionExecutor-1626-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
   [junit4]   2> 592932 DEBUG (AutoscalingActionExecutor-1626-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 51 ms for event id=568cb56fb1bc58Tbecttotktkl746gglgm51fi0z
   [junit4]   2> 592982 INFO  (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.TestTriggerIntegration ====== KILL OVERSEER 2
   [junit4]   2> 592982 INFO  (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 592982 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 592982 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 592982 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 592982 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 592982 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 592984 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 592984 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 592998 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
   [junit4]   2> 592999 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 592999 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 592999 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion -1
   [junit4]   2> 592999 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 592999 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_trigger - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10011_solr]
   [junit4]   2> 592999 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:10008_solr
   [junit4]   2> 593018 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10011_solr]
   [junit4]   2> 593018 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventMarkerAction}], enabled=true}
   [junit4]   2> 593022 DEBUG (ScheduledTrigger-1628-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
   [junit4]   2> 593022 DEBUG (ScheduledTrigger-1628-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10008_solr
   [junit4]   2> 593038 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10011_solr]
   [junit4]   2> 593039 DEBUG (ScheduledTrigger-1628-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 593039 DEBUG (ScheduledTrigger-1628-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 593039 DEBUG (ScheduledTrigger-1628-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 593039 DEBUG (ScheduledTrigger-1628-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10008_solr
   [junit4]   2> 593050 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 593065 DEBUG (ScheduledTrigger-1628-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
   [junit4]   2> 593065 DEBUG (ScheduledTrigger-1628-thread-2) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10008_solr]
   [junit4]   2> 593065 DEBUG (ScheduledTrigger-1628-thread-2) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"568cb713e089b6Tbecttotktkl746gglgm51fi12",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":24361565937502646,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[24361565937502646],
   [junit4]   2>     "nodeNames":["127.0.0.1:10008_solr"]}}
   [junit4]   2> 593065 DEBUG (ScheduledTrigger-1628-thread-2) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_lost_trigger, node_added_trigger]
   [junit4]   2> 593083 DEBUG (AutoscalingActionExecutor-1629-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"568cb713e089b6Tbecttotktkl746gglgm51fi12",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":24361565937502646,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[24361565937502646],
   [junit4]   2>     "_enqueue_time_":24361568074777046,
   [junit4]   2>     "nodeNames":["127.0.0.1:10008_solr"]}}
   [junit4]   2> 593089 INFO  (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeMarkersRegistration
   [junit4]   2> 593090 INFO  (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes: 2
   [junit4]   2> ## Empty nodes: 2
   [junit4]   2> ## Dead nodes: 2
   [junit4]   2> ## 127.0.0.1:10008_solr
   [junit4]   2> ## 127.0.0.1:10009_solr
   [junit4]   2> ## Collections: []
   [junit4]   2> ## Max replicas per node: 0
   [junit4]   2> ## Min replicas per node: 0
   [junit4]   2> ## Total replicas: 0
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ## - autoscaling        3
   [junit4]   2> ## - update             3
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: node_added_trigger
   [junit4]   2> ## - STARTED         1
   [junit4]   2> ## - SUCCEEDED       1
   [junit4]   2> ## * Trigger: node_lost_trigger
   [junit4]   2> ## - STARTED         1
   [junit4]   2>
   [junit4]   2> 593206 DEBUG (AutoscalingActionExecutor-1629-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 593206 DEBUG (AutoscalingActionExecutor-1629-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger after 100ms
   [junit4]   2> 593206 DEBUG (AutoscalingActionExecutor-1629-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
   [junit4]   2> 593206 DEBUG (AutoscalingActionExecutor-1629-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 6186 ms for event id=568cb713e089b6Tbecttotktkl746gglgm51fi12
   [junit4]   2> 593306 DEBUG (ScheduledTrigger-1628-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
   [junit4]   2> 593327 DEBUG (ScheduledTrigger-1628-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
   [junit4]   2> 593330 DEBUG (ScheduledTrigger-1628-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 593330 DEBUG (ScheduledTrigger-1628-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 593330 DEBUG (ScheduledTrigger-1628-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 593332 INFO  (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTriggerRestoreState
   [junit4]   2> 593332 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
   [junit4]   2> 593332 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
   [junit4]   2> 593332 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10009_solr
   [junit4]   2> 593332 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10008_solr
   [junit4]   2> 593332 INFO  (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 593332 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 593332 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 593332 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 593332 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 593332 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 593333 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 593335 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 593335 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 593332 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_152]
   [junit4]   2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_152]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:169) [java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
   [junit4]   2> 593344 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 593345 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 11
   [junit4]   2> 593345 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion -1
   [junit4]   2> 593345 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
   [junit4]   2> 593374 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10011_solr]
   [junit4]   2> 593382 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 593383 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion 11
   [junit4]   2> 593383 DEBUG (ScheduledTrigger-1631-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 593403 DEBUG (ScheduledTrigger-1631-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 593423 DEBUG (ScheduledTrigger-1631-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 593444 DEBUG (ScheduledTrigger-1631-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 593459 DEBUG (simCloudManagerPool-1630-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 593459 DEBUG (simCloudManagerPool-1630-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 12
   [junit4]   2> 593460 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 12
   [junit4]   2> 593460 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 593460 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 12, lastZnodeVersion 12
   [junit4]   2> 593465 DEBUG (ScheduledTrigger-1631-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 593475 DEBUG (simCloudManagerPool-1630-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 593475 DEBUG (simCloudManagerPool-1630-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 13
   [junit4]   2> 593476 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 13
   [junit4]   2> 593476 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 593476 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10012_solr, 127.0.0.1:10011_solr]
   [junit4]   2> 593476 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 593476 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion 13
   [junit4]   2> 593476 DEBUG (ScheduledTrigger-1631-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 3
   [junit4]   2> 593485 DEBUG (ScheduledTrigger-1631-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 593485 DEBUG (ScheduledTrigger-1631-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10012_solr
   [junit4]   2> 593496 DEBUG (ScheduledTrigger-1631-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2
   [junit4]   2> 593497 DEBUG (ScheduledTrigger-1631-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10012_solr
   [junit4]   2> 593505 DEBUG (ScheduledTrigger-1631-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 593517 DEBUG (ScheduledTrigger-1631-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2
   [junit4]   2> 593518 DEBUG (simCloudManagerPool-1630-thread-3) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 593518 DEBUG (simCloudManagerPool-1630-thread-3) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 14
   [junit4]   2> 593518 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 14
   [junit4]   2> 593522 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 593523 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10011_solr]
   [junit4]   2> 593523 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:10012_solr
   [junit4]   2> 593523 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 593523 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 14, lastZnodeVersion 14
   [junit4]   2> 593523 DEBUG (ScheduledTrigger-1631-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2
   [junit4]   2> 593523 DEBUG (ScheduledTrigger-1631-thread-2) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10012_solr]
   [junit4]   2> 593523 DEBUG (ScheduledTrigger-1631-thread-2) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"568cbce6b72e62Tbecttotktkl746gglgm51fi15",
   [junit4]   2>   "source":"node_lost_restore_trigger",
   [junit4]   2>   "eventTime":24361590949621346,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[24361590949621346],
   [junit4]   2>     "nodeNames":["127.0.0.1:10012_solr"]}}
   [junit4]   2> 593523 DEBUG (ScheduledTrigger-1631-thread-2) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_lost_restore_trigger]
   [junit4]   2> 593535 DEBUG (AutoscalingActionExecutor-1632-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"568cbce6b72e62Tbecttotktkl746gglgm51fi15",
   [junit4]   2>   "source":"node_lost_restore_trigger",
   [junit4]   2>   "eventTime":24361590949621346,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[24361590949621346],
   [junit4]   2>     "_enqueue_time_":24361590980493396,
   [junit4]   2>     "nodeNames":["127.0.0.1:10012_solr"]}}
   [junit4]   2> 593536 INFO  (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTriggerRestoreState
   [junit4]   2> 593536 INFO  (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes: 2
   [junit4]   2> ## Empty nodes: 2
   [junit4]   2> ## Dead nodes: 1
   [junit4]   2> ## 127.0.0.1:10012_solr
   [junit4]   2> ## Collections: []
   [junit4]   2> ## Max replicas per node: 0
   [junit4]   2> ## Min replicas per node: 0
   [junit4]   2> ## Total replicas: 0
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ## - autoscaling        3
   [junit4]   2> ## - update             1
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: node_lost_restore_trigger
   [junit4]   2> ## - STARTED         1
   [junit4]   2>
   [junit4]   2> 593561 INFO  (TEST-TestTriggerIntegration.testListeners-seed#[F4FDB65561595C5B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testListeners
   [junit4]   2> 593561 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 15
   [junit4]   2> 593561 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10012_solr
   [junit4]   2> 593561 INFO  (TEST-TestTriggerIntegration.testListeners-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 593562 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 593566 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 593566 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 593566 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 593567 WARN  (AutoscalingActionExecutor-1632-thread-1) [    ] o.a.s.c.a.SystemLogListener Exception sending event to collection .system
   [junit4]   2> java.io.IOException: java.lang.InterruptedException
   [junit4]   2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:540) ~[test/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.SystemLogListener.onEvent(SystemLogListener.java:118) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:790) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:757) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:322) ~[java/:?]
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_152]
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_152]
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:192) ~[java/:?]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
   [junit4]   2> Caused by: java.lang.InterruptedException
   [junit4]   2> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) ~[?:1.8.0_152]
   [junit4]   2> at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:1.8.0_152]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:538) ~[test/:?]
   [junit4]   2> ... 10 more
   [junit4]   2> 593567 DEBUG (AutoscalingActionExecutor-1632-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 1596 ms for event id=568cbce6b72e62Tbecttotktkl746gglgm51fi15
   [junit4]   2> 593568 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 593568 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 593568 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 593576 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 593576 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 16
   [junit4]   2> 593576 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 16, lastZnodeVersion -1
   [junit4]   2> 593576 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 16
   [junit4]   2> 593582 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10011_solr]
   [junit4]   2> 593594 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 593594 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 16, lastZnodeVersion 16
   [junit4]   2> 593595 DEBUG (ScheduledTrigger-1634-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 593712 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 593713 DEBUG (simCloudManagerPool-1633-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 593714 DEBUG (simCloudManagerPool-1633-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 17
   [junit4]   2> 593719 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 17
   [junit4]   2> 593719 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 593719 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 17, lastZnodeVersion 17
   [junit4]   2> 593731 DEBUG (simCloudManagerPool-1633-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 593732 DEBUG (simCloudManagerPool-1633-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 18
   [junit4]   2> 593732 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 18
   [junit4]   2> 593732 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 593732 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10011_solr]
   [junit4]   2> 593733 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}, {name=test1, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction}], enabled=true}
   [junit4]   2> 593733 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 18, lastZnodeVersion 18
   [junit4]   2> 593733 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 593733 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 593733 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 593745 DEBUG (simCloudManagerPool-1633-thread-3) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 593745 DEBUG (simCloudManagerPool-1633-thread-3) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 19
   [junit4]   2> 593746 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 19
   [junit4]   2> 593746 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 19, lastZnodeVersion 19
   [junit4]   2> 593753 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 593753 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 593753 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 593754 DEBUG (simCloudManagerPool-1633-thread-4) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 593754 DEBUG (simCloudManagerPool-1633-thread-4) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 20
   [junit4]   2> 593755 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 20
   [junit4]   2> 593755 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 20, lastZnodeVersion 20
   [junit4]   2> 593773 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 593773 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 593773 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 593773 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10013_solr at time 24361603492311396
   [junit4]   2> 593794 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 593794 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 593794 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 593814 DEBUG (ScheduledTrigger-1634-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 593814 DEBUG (ScheduledTrigger-1634-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 593814 DEBUG (ScheduledTrigger-1634-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 593815 DEBUG (ScheduledTrigger-1634-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10013_solr] added at times [24361603492311396], now=24361605546146146
   [junit4]   2> 593815 DEBUG (ScheduledTrigger-1634-thread-3) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"568cbfd2517164Tbecttotktkl746gglgm51fi18",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":24361603492311396,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[24361603492311396],
   [junit4]   2>     "nodeNames":["127.0.0.1:10013_solr"]}}
   [junit4]   2> 593815 DEBUG (ScheduledTrigger-1634-thread-3) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_trigger]
   [junit4]   2> 593834 DEBUG (ScheduledTrigger-1634-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - removing marker path: /autoscaling/nodeAdded/127.0.0.1:10013_solr
   [junit4]   2> 593834 DEBUG (AutoscalingActionExecutor-1635-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"568cbfd2517164Tbecttotktkl746gglgm51fi18",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":24361603492311396,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[24361603492311396],
   [junit4]   2>     "_enqueue_time_":24361605588831696,
   [junit4]   2>     "nodeNames":["127.0.0.1:10013_solr"]}}
   [junit4]   2> 593854 DEBUG (AutoscalingActionExecutor-1635-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 593854 DEBUG (AutoscalingActionExecutor-1635-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
   [junit4]   2> 593854 DEBUG (AutoscalingActionExecutor-1635-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 1001 ms for event id=568cbfd2517164Tbecttotktkl746gglgm51fi18
   [junit4]   2> 593954 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
   [junit4]   2> 593954 DEBUG (ScheduledTrigger-1634-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 593955 DEBUG (ScheduledTrigger-1634-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 593955 DEBUG (ScheduledTrigger-1634-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10014_solr at time 24361612551539496
   [junit4]   2> 593975 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
   [junit4]   2> 593975 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 593975 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 593995 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
   [junit4]   2> 593995 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 593995 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 593995 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10014_solr] added at times [24361612551539496], now=24361614590599346
   [junit4]   2> 593996 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"568cc1ee4a4b28Tbecttotktkl746gglgm51fi1b",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":24361612551539496,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[24361612551539496],
   [junit4]   2>     "nodeNames":["127.0.0.1:10014_solr"]}}
   [junit4]   2> 593996 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_trigger]
   [junit4]   2> 594006 DEBUG (ScheduledTrigger-1634-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - removing marker path: /autoscaling/nodeAdded/127.0.0.1:10014_solr
   [junit4]   2> 594007 DEBUG (AutoscalingActionExecutor-1635-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"568cc1ee4a4b28Tbecttotktkl746gglgm51fi1b",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":24361612551539496,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[24361612551539496],
   [junit4]   2>     "_enqueue_time_":24361614604136746,
   [junit4]   2>     "nodeNames":["127.0.0.1:10014_solr"]}}
   [junit4]   2> 594015 WARN  (AutoscalingActionExecutor-1635-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Exception executing actions
   [junit4]   2> java.lang.Exception: Error executing action: test1 for trigger event: {
   [junit4]   2>   "id":"568cc1ee4a4b28Tbecttotktkl746gglgm51fi1b",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":24361612551539496,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[24361612551539496],
   [junit4]   2>     "_enqueue_time_":24361614604136746,
   [junit4]   2>     "nodeNames":["127.0.0.1:10014_solr"]}}
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:312) ~[java/:?]
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_152]
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_152]
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:192) ~[java/:?]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
   [junit4]   2> Caused by: java.lang.RuntimeException: failure
   [junit4]   2> at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction.process(TestTriggerIntegration.java:909) ~[test/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:309) ~[java/:?]
   [junit4]   2> ... 6 more
   [junit4]   2> 594016 DEBUG (AutoscalingActionExecutor-1635-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 594016 DEBUG (AutoscalingActionExecutor-1635-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
   [junit4]   2> 594016 DEBUG (AutoscalingActionExecutor-1635-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 471 ms for event id=568cc1ee4a4b28Tbecttotktkl746gglgm51fi1b
   [junit4]   2> 594048 INFO  (TEST-TestTriggerIntegration.testListeners-seed#[F4FDB65561595C5B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testListeners
   [junit4]   2> 594049 INFO  (TEST-TestTriggerIntegration.testListeners-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes: 4
   [junit4]   2> ## Empty nodes: 4
   [junit4]   2> ## Dead nodes: 0
   [junit4]   2> ## Collections: []
   [junit4]   2> ## Max replicas per node: 0
   [junit4]   2> ## Min replicas per node: 0
   [junit4]   2> ## Total replicas: 0
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ## - autoscaling        4
   [junit4]   2> ## - update             4
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: node_added_trigger
   [junit4]   2> ## - FAILED          1
   [junit4]   2> ## - STARTED         2
   [junit4]   2> ## - SUCCEEDED       1
   [junit4]   2> ## * Trigger: node_lost_restore_trigger
   [junit4]   2> ## - SUCCEEDED       1
   [junit4]   2>
   [junit4]   2> 594052 INFO  (TEST-TestTriggerIntegration.testEventQueue-seed#[F4FDB65561595C5B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testEventQueue
   [junit4]   2> 594053 DEBUG (TEST-TestTriggerIntegration.testEventQueue-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 21
   [junit4]   2> 594053 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 21
   [junit4]   2> 594053 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 594053 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 594053 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 21, lastZnodeVersion 21
   [junit4]   2> 594053 INFO  (TEST-TestTriggerIntegration.testEventQueue-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 594053 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_152]
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_152]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:181) [java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
   [junit4]   2> 594053 DEBUG (TEST-TestTriggerIntegration.testEventQueue-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 594054 DEBUG (TEST-TestTriggerIntegration.testEventQueue-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 594054 DEBUG (TEST-TestTriggerIntegration.testEventQueue-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 594054 DEBUG (TEST-TestTriggerIntegration.testEventQueue-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 594054 DEBUG (TEST-TestTriggerIntegration.testEventQueue-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 594054 DEBUG (TEST-TestTriggerIntegration.testEventQueue-seed#[F4FDB65561595C5B]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 594074 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 594075 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 22
   [junit4]   2> 594075 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 22, lastZnodeVersion -1
   [junit4]   2> 594075 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 22
   [junit4]   2> 594076 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr]
   [junit4]   2> 594111 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 594111 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 22, lastZnodeVersion 22
   [junit4]   2> 594111 DEBUG (ScheduledTrigger-1637-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 594132 DEBUG (ScheduledTrigger-1637-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 594153 DEBUG (ScheduledTrigger-1637-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 594173 DEBUG (ScheduledTrigger-1637-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 594191 DEBUG (simCloudManagerPool-1636-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 594191 DEBUG (simCloudManagerPool-1636-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 23
   [junit4]   2> 594192 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 23
   [junit4]   2> 594192 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 594192 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 23, lastZnodeVersion 23
   [junit4]   2> 594193 DEBUG (ScheduledTrigger-1637-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 594194 INFO  (simCloudManagerPool-1636-thread-2) [    ] o.a.s.c.a.s.TestTriggerIntegration TestEventQueueAction instantiated
   [junit4]   2> 594194 DEBUG (simCloudManagerPool-1636-thread-2) [    ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 594194 DEBUG (simCloudManagerPool-1636-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 594194 DEBUG (simCloudManagerPool-1636-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 24
   [junit4]   2> 594195 INFO  (simCloudManagerPool-1636-thread-2) [    ] o.a.s.c.a.s.TestTriggerIntegration TestEventQueueAction instantiated
   [junit4]   2> 594195 DEBUG (simCloudManagerPool-1636-thread-2) [    ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 594195 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 24
   [junit4]   2> 594195 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr]
   [junit4]   2> 594195 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventQueueAction}], enabled=true}
   [junit4]   2> 594195 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 24, lastZnodeVersion 24
   [junit4]   2> 594195 DEBUG (ScheduledTrigger-1637-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 594195 DEBUG (ScheduledTrigger-1637-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 594213 DEBUG (ScheduledTrigger-1637-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 594215 DEBUG (ScheduledTrigger-1637-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 594215 DEBUG (ScheduledTrigger-1637-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 594216 DEBUG (ScheduledTrigger-1637-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10015_solr at time 24361625594293396
   [junit4]   2> 594233 DEBUG (ScheduledTrigger-1637-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 594236 DEBUG (ScheduledTrigger-1637-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 594236 DEBUG (ScheduledTrigger-1637-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 594236 DEBUG (ScheduledTrigger-1637-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing registered processor for nodes: [127.0.0.1:10015_solr] added at times [24361625594293396], now=24361626608471546
   [junit4]   2> 594236 DEBUG (ScheduledTrigger-1637-thread-3) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"568cc4f7b2ec94Tbecttotktkl746gglgm51fi1e",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":24361625594293396,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[24361625594293396],
   [junit4]   2>     "nodeNames":["127.0.0.1:10015_solr"]}}
   [junit4]   2> 594236 DEBUG (ScheduledTrigger-1637-thread-3) [    ] o.a.s.c.a.S

[...truncated too long message...]

d-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null1928908922
     [copy] Copying 40 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null1928908922
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null1928908922

resolve-example:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve-server:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1459724795
     [copy] Copying 246 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1459724795
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1459724795

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 89ms :: artifacts dl 10ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 126 minutes 2 seconds
Archiving artifacts
WARN: No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
WARN: java.lang.InterruptedException: no matches found within 10000
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]