[JENKINS] Lucene-Solr-BadApples-NightlyTests-7.x - Build # 33 - Failure

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-NightlyTests-7.x - Build # 33 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-7.x/33/

11 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:36095/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:36095/collection1
        at __randomizedtesting.SeedInfo.seed([27819D609D1F050:8A2C260CA72D9DA8]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:504)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:479)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:1590)
        at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:215)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 53 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest:     1) Thread[id=11700, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[27819D609D1F050]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)    2) Thread[id=11701, name=zkConnectionManagerCallback-3392-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=11699, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[27819D609D1F050]-SendThread(127.0.0.1:33747), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)    4) Thread[id=11698, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest:
   1) Thread[id=11700, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[27819D609D1F050]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
   2) Thread[id=11701, name=zkConnectionManagerCallback-3392-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=11699, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[27819D609D1F050]-SendThread(127.0.0.1:33747), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
   4) Thread[id=11698, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([27819D609D1F050]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=11701, name=zkConnectionManagerCallback-3392-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=11699, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[27819D609D1F050]-SendThread(127.0.0.1:33747), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=11701, name=zkConnectionManagerCallback-3392-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=11699, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[27819D609D1F050]-SendThread(127.0.0.1:33747), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)
        at __randomizedtesting.SeedInfo.seed([27819D609D1F050]:0)


FAILED:  org.apache.solr.cloud.api.collections.TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete

Error Message:
Error from server at http://127.0.0.1:43320/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/testcollection_shard1_replica_n3/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason: <pre>    Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>  </body> </html>

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://127.0.0.1:43320/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/testcollection_shard1_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at __randomizedtesting.SeedInfo.seed([27819D609D1F050:A182B7738E391AF5]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:237)
        at org.apache.solr.cloud.api.collections.TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete(TestCollectionsAPIViaSolrCloudCluster.java:170)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:43320/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/testcollection_shard1_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more


FAILED:  org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup

Error Message:
missing cleanup event: [CapturedEvent{timestamp=2072443178755382, stage=STARTED, actionName='null', event={   "id":"75ce047444daaT5p58qo0hbelg2vmyg1fy6rtxb",   "source":".scheduled_maintenance",   "eventTime":2072443175062954,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937657402,     "_enqueue_time_":2072443176907937}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072443180737590, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={   "id":"75ce047444daaT5p58qo0hbelg2vmyg1fy6rtxb",   "source":".scheduled_maintenance",   "eventTime":2072443175062954,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937657402,     "_enqueue_time_":2072443176907937}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072443198284716, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={   "id":"75ce047444daaT5p58qo0hbelg2vmyg1fy6rtxb",   "source":".scheduled_maintenance",   "eventTime":2072443175062954,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937657402,     "_enqueue_time_":2072443176907937}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.inactive_shard_plan={staleLocks={ScheduledMaintenanceTriggerTest_collection1/staleShard-splitting={stateTimestamp=1540764857375462512, currentTimeNs=1540937657424373220, deltaSec=172800, ttlSec=20}}}, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072443201434624, stage=SUCCEEDED, actionName='null', event={   "id":"75ce047444daaT5p58qo0hbelg2vmyg1fy6rtxb",   "source":".scheduled_maintenance",   "eventTime":2072443175062954,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937657402,     "_enqueue_time_":2072443176907937}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072448204734725, stage=STARTED, actionName='null', event={   "id":"75ce1730008abT5p58qo0hbelg2vmyg1fy6rtxh",   "source":".scheduled_maintenance",   "eventTime":2072448203753643,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937662431,     "_enqueue_time_":2072448203980419}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072448205469030, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={   "id":"75ce1730008abT5p58qo0hbelg2vmyg1fy6rtxh",   "source":".scheduled_maintenance",   "eventTime":2072448203753643,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937662431,     "_enqueue_time_":2072448203980419}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072448209499488, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={   "id":"75ce1730008abT5p58qo0hbelg2vmyg1fy6rtxh",   "source":".scheduled_maintenance",   "eventTime":2072448203753643,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937662431,     "_enqueue_time_":2072448203980419}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072448210982445, stage=SUCCEEDED, actionName='null', event={   "id":"75ce1730008abT5p58qo0hbelg2vmyg1fy6rtxh",   "source":".scheduled_maintenance",   "eventTime":2072448203753643,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937662431,     "_enqueue_time_":2072448203980419}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072453212386765, stage=STARTED, actionName='null', event={   "id":"75ce29d7bdb1bT5p58qo0hbelg2vmyg1fy6rtxi",   "source":".scheduled_maintenance",   "eventTime":2072453211478811,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937667439,     "_enqueue_time_":2072453211731127}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072453213083124, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={   "id":"75ce29d7bdb1bT5p58qo0hbelg2vmyg1fy6rtxi",   "source":".scheduled_maintenance",   "eventTime":2072453211478811,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937667439,     "_enqueue_time_":2072453211731127}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072453215766094, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={   "id":"75ce29d7bdb1bT5p58qo0hbelg2vmyg1fy6rtxi",   "source":".scheduled_maintenance",   "eventTime":2072453211478811,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937667439,     "_enqueue_time_":2072453211731127}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072453217129932, stage=SUCCEEDED, actionName='null', event={   "id":"75ce29d7bdb1bT5p58qo0hbelg2vmyg1fy6rtxi",   "source":".scheduled_maintenance",   "eventTime":2072453211478811,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937667439,     "_enqueue_time_":2072453211731127}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072458218763335, stage=STARTED, actionName='null', event={   "id":"75ce3c7e16cf7T5p58qo0hbelg2vmyg1fy6rtxj",   "source":".scheduled_maintenance",   "eventTime":2072458217745655,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937672445,     "_enqueue_time_":2072458218015497}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072458219652691, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={   "id":"75ce3c7e16cf7T5p58qo0hbelg2vmyg1fy6rtxj",   "source":".scheduled_maintenance",   "eventTime":2072458217745655,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937672445,     "_enqueue_time_":2072458218015497}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072458222208788, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={   "id":"75ce3c7e16cf7T5p58qo0hbelg2vmyg1fy6rtxj",   "source":".scheduled_maintenance",   "eventTime":2072458217745655,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937672445,     "_enqueue_time_":2072458218015497}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072458223619204, stage=SUCCEEDED, actionName='null', event={   "id":"75ce3c7e16cf7T5p58qo0hbelg2vmyg1fy6rtxj",   "source":".scheduled_maintenance",   "eventTime":2072458217745655,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937672445,     "_enqueue_time_":2072458218015497}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072463225175175, stage=STARTED, actionName='null', event={   "id":"75ce4f249e208T5p58qo0hbelg2vmyg1fy6rtxk",   "source":".scheduled_maintenance",   "eventTime":2072463224201736,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937677451,     "_enqueue_time_":2072463224445698}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072463225893312, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={   "id":"75ce4f249e208T5p58qo0hbelg2vmyg1fy6rtxk",   "source":".scheduled_maintenance",   "eventTime":2072463224201736,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937677451,     "_enqueue_time_":2072463224445698}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072463228827718, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={   "id":"75ce4f249e208T5p58qo0hbelg2vmyg1fy6rtxk",   "source":".scheduled_maintenance",   "eventTime":2072463224201736,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937677451,     "_enqueue_time_":2072463224445698}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072463230727086, stage=SUCCEEDED, actionName='null', event={   "id":"75ce4f249e208T5p58qo0hbelg2vmyg1fy6rtxk",   "source":".scheduled_maintenance",   "eventTime":2072463224201736,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937677451,     "_enqueue_time_":2072463224445698}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072468232425669, stage=STARTED, actionName='null', event={   "id":"75ce61cbc5d7aT5p58qo0hbelg2vmyg1fy6rtxl",   "source":".scheduled_maintenance",   "eventTime":2072468231314810,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937682458,     "_enqueue_time_":2072468231602025}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072468233201881, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={   "id":"75ce61cbc5d7aT5p58qo0hbelg2vmyg1fy6rtxl",   "source":".scheduled_maintenance",   "eventTime":2072468231314810,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937682458,     "_enqueue_time_":2072468231602025}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072468235525545, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={   "id":"75ce61cbc5d7aT5p58qo0hbelg2vmyg1fy6rtxl",   "source":".scheduled_maintenance",   "eventTime":2072468231314810,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1540937682458,     "_enqueue_time_":2072468231602025}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.inactive_shard_plan={staleLocks={ScheduledMaintenanceTriggerTest_collection1/shard1-splitting={stateTimestamp=1540937657422126363, currentTimeNs=1540937682462216654, deltaSec=25, ttlSec=20}}}, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}]

Stack Trace:
java.lang.AssertionError: missing cleanup event: [CapturedEvent{timestamp=2072443178755382, stage=STARTED, actionName='null', event={
  "id":"75ce047444daaT5p58qo0hbelg2vmyg1fy6rtxb",
  "source":".scheduled_maintenance",
  "eventTime":2072443175062954,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937657402,
    "_enqueue_time_":2072443176907937}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072443180737590, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={
  "id":"75ce047444daaT5p58qo0hbelg2vmyg1fy6rtxb",
  "source":".scheduled_maintenance",
  "eventTime":2072443175062954,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937657402,
    "_enqueue_time_":2072443176907937}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072443198284716, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={
  "id":"75ce047444daaT5p58qo0hbelg2vmyg1fy6rtxb",
  "source":".scheduled_maintenance",
  "eventTime":2072443175062954,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937657402,
    "_enqueue_time_":2072443176907937}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.inactive_shard_plan={staleLocks={ScheduledMaintenanceTriggerTest_collection1/staleShard-splitting={stateTimestamp=1540764857375462512, currentTimeNs=1540937657424373220, deltaSec=172800, ttlSec=20}}}, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072443201434624, stage=SUCCEEDED, actionName='null', event={
  "id":"75ce047444daaT5p58qo0hbelg2vmyg1fy6rtxb",
  "source":".scheduled_maintenance",
  "eventTime":2072443175062954,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937657402,
    "_enqueue_time_":2072443176907937}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072448204734725, stage=STARTED, actionName='null', event={
  "id":"75ce1730008abT5p58qo0hbelg2vmyg1fy6rtxh",
  "source":".scheduled_maintenance",
  "eventTime":2072448203753643,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937662431,
    "_enqueue_time_":2072448203980419}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072448205469030, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={
  "id":"75ce1730008abT5p58qo0hbelg2vmyg1fy6rtxh",
  "source":".scheduled_maintenance",
  "eventTime":2072448203753643,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937662431,
    "_enqueue_time_":2072448203980419}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072448209499488, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={
  "id":"75ce1730008abT5p58qo0hbelg2vmyg1fy6rtxh",
  "source":".scheduled_maintenance",
  "eventTime":2072448203753643,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937662431,
    "_enqueue_time_":2072448203980419}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072448210982445, stage=SUCCEEDED, actionName='null', event={
  "id":"75ce1730008abT5p58qo0hbelg2vmyg1fy6rtxh",
  "source":".scheduled_maintenance",
  "eventTime":2072448203753643,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937662431,
    "_enqueue_time_":2072448203980419}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072453212386765, stage=STARTED, actionName='null', event={
  "id":"75ce29d7bdb1bT5p58qo0hbelg2vmyg1fy6rtxi",
  "source":".scheduled_maintenance",
  "eventTime":2072453211478811,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937667439,
    "_enqueue_time_":2072453211731127}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072453213083124, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={
  "id":"75ce29d7bdb1bT5p58qo0hbelg2vmyg1fy6rtxi",
  "source":".scheduled_maintenance",
  "eventTime":2072453211478811,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937667439,
    "_enqueue_time_":2072453211731127}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072453215766094, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={
  "id":"75ce29d7bdb1bT5p58qo0hbelg2vmyg1fy6rtxi",
  "source":".scheduled_maintenance",
  "eventTime":2072453211478811,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937667439,
    "_enqueue_time_":2072453211731127}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072453217129932, stage=SUCCEEDED, actionName='null', event={
  "id":"75ce29d7bdb1bT5p58qo0hbelg2vmyg1fy6rtxi",
  "source":".scheduled_maintenance",
  "eventTime":2072453211478811,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937667439,
    "_enqueue_time_":2072453211731127}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072458218763335, stage=STARTED, actionName='null', event={
  "id":"75ce3c7e16cf7T5p58qo0hbelg2vmyg1fy6rtxj",
  "source":".scheduled_maintenance",
  "eventTime":2072458217745655,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937672445,
    "_enqueue_time_":2072458218015497}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072458219652691, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={
  "id":"75ce3c7e16cf7T5p58qo0hbelg2vmyg1fy6rtxj",
  "source":".scheduled_maintenance",
  "eventTime":2072458217745655,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937672445,
    "_enqueue_time_":2072458218015497}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072458222208788, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={
  "id":"75ce3c7e16cf7T5p58qo0hbelg2vmyg1fy6rtxj",
  "source":".scheduled_maintenance",
  "eventTime":2072458217745655,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937672445,
    "_enqueue_time_":2072458218015497}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072458223619204, stage=SUCCEEDED, actionName='null', event={
  "id":"75ce3c7e16cf7T5p58qo0hbelg2vmyg1fy6rtxj",
  "source":".scheduled_maintenance",
  "eventTime":2072458217745655,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937672445,
    "_enqueue_time_":2072458218015497}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072463225175175, stage=STARTED, actionName='null', event={
  "id":"75ce4f249e208T5p58qo0hbelg2vmyg1fy6rtxk",
  "source":".scheduled_maintenance",
  "eventTime":2072463224201736,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937677451,
    "_enqueue_time_":2072463224445698}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072463225893312, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={
  "id":"75ce4f249e208T5p58qo0hbelg2vmyg1fy6rtxk",
  "source":".scheduled_maintenance",
  "eventTime":2072463224201736,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937677451,
    "_enqueue_time_":2072463224445698}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072463228827718, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={
  "id":"75ce4f249e208T5p58qo0hbelg2vmyg1fy6rtxk",
  "source":".scheduled_maintenance",
  "eventTime":2072463224201736,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937677451,
    "_enqueue_time_":2072463224445698}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072463230727086, stage=SUCCEEDED, actionName='null', event={
  "id":"75ce4f249e208T5p58qo0hbelg2vmyg1fy6rtxk",
  "source":".scheduled_maintenance",
  "eventTime":2072463224201736,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937677451,
    "_enqueue_time_":2072463224445698}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072468232425669, stage=STARTED, actionName='null', event={
  "id":"75ce61cbc5d7aT5p58qo0hbelg2vmyg1fy6rtxl",
  "source":".scheduled_maintenance",
  "eventTime":2072468231314810,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937682458,
    "_enqueue_time_":2072468231602025}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072468233201881, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={
  "id":"75ce61cbc5d7aT5p58qo0hbelg2vmyg1fy6rtxl",
  "source":".scheduled_maintenance",
  "eventTime":2072468231314810,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937682458,
    "_enqueue_time_":2072468231602025}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=2072468235525545, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={
  "id":"75ce61cbc5d7aT5p58qo0hbelg2vmyg1fy6rtxl",
  "source":".scheduled_maintenance",
  "eventTime":2072468231314810,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1540937682458,
    "_enqueue_time_":2072468231602025}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.inactive_shard_plan={staleLocks={ScheduledMaintenanceTriggerTest_collection1/shard1-splitting={stateTimestamp=1540937657422126363, currentTimeNs=1540937682462216654, deltaSec=25, ttlSec=20}}}, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}]
        at __randomizedtesting.SeedInfo.seed([27819D609D1F050:1F54D9A46892D75B]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup(ScheduledMaintenanceTriggerTest.java:253)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:38640/x_okt/i/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:38640/x_okt/i/collection1
        at __randomizedtesting.SeedInfo.seed([27819D609D1F050:8A2C260CA72D9DA8]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:504)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:479)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:1590)
        at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:215)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 53 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest:     1) Thread[id=17993, name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[27819D609D1F050]-EventThread, state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)    2) Thread[id=17992, name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[27819D609D1F050]-SendThread(127.0.0.1:40136), state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)    3) Thread[id=17994, name=zkConnectionManagerCallback-3466-thread-1, state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=17991, name=Connection evictor, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest:
   1) Thread[id=17993, name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[27819D609D1F050]-EventThread, state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
   2) Thread[id=17992, name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[27819D609D1F050]-SendThread(127.0.0.1:40136), state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
   3) Thread[id=17994, name=zkConnectionManagerCallback-3466-thread-1, state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=17991, name=Connection evictor, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([27819D609D1F050]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=17992, name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[27819D609D1F050]-SendThread(127.0.0.1:40136), state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)    2) Thread[id=17994, name=zkConnectionManagerCallback-3466-thread-1, state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=17992, name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[27819D609D1F050]-SendThread(127.0.0.1:40136), state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)
   2) Thread[id=17994, name=zkConnectionManagerCallback-3466-thread-1, state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([27819D609D1F050]:0)


FAILED:  org.apache.solr.cloud.hdfs.HdfsRestartWhileUpdatingTest.test

Error Message:
There are still nodes recoverying - waited for 320 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 320 seconds
        at __randomizedtesting.SeedInfo.seed([27819D609D1F050:8A2C260CA72D9DA8]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:185)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:920)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1477)
        at org.apache.solr.cloud.RestartWhileUpdatingTest.test(RestartWhileUpdatingTest.java:145)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at __randomizedtesting.SeedInfo.seed([27819D609D1F050:31CA311204662AE7]:0)
        at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:84)
        at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:57)
        at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:168)
        at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:154)
        at org.apache.lucene.store.DataOutput.copyBytes(DataOutput.java:278)
        at org.apache.lucene.store.MockIndexOutputWrapper.copyBytes(MockIndexOutputWrapper.java:165)
        at org.apache.lucene.codecs.simpletext.SimpleTextCompoundFormat.write(SimpleTextCompoundFormat.java:186)
        at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4997)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4510)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068)
        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2170)
        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5103)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1612)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1228)
        at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:189)
        at org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit(TestDocTermOrdsUninvertLimit.java:66)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testVersionsAreReturned

Error Message:
Error from server at https://127.0.0.1:41935/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason: <pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>  </body> </html>

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://127.0.0.1:41935/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at __randomizedtesting.SeedInfo.seed([9919E1BF1324B5F4:61DF188A803D2D3C]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:237)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testVersionsAreReturned(CloudSolrClientTest.java:725)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apa

[...truncated too long message...]


   [junit4]   1> </body>
   [junit4]   1> </html>
   [junit4]   1> , retry? 3
   [junit4]   1> 22:26:38.424 [TEST-CloudSolrClientTest.testVersionsAreReturned-seed#[9919E1BF1324B5F4]] ERROR org.apache.solr.client.solrj.impl.CloudSolrClient - Request to collection [collection1] failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:41935/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]   1> <head>
   [junit4]   1> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
   [junit4]   1> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
   [junit4]   1> </head>
   [junit4]   1> <body><h2>HTTP ERROR 404</h2>
   [junit4]   1> <p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
   [junit4]   1> <pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>
   [junit4]   1>
   [junit4]   1> </body>
   [junit4]   1> </html>
   [junit4]   1> , retry? 4
   [junit4]   1> 22:26:38.430 [TEST-CloudSolrClientTest.testVersionsAreReturned-seed#[9919E1BF1324B5F4]] ERROR org.apache.solr.client.solrj.impl.CloudSolrClient - Request to collection [collection1] failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:41935/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]   1> <head>
   [junit4]   1> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
   [junit4]   1> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
   [junit4]   1> </head>
   [junit4]   1> <body><h2>HTTP ERROR 404</h2>
   [junit4]   1> <p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
   [junit4]   1> <pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>
   [junit4]   1>
   [junit4]   1> </body>
   [junit4]   1> </html>
   [junit4]   1> , retry? 5
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrClientTest -Dtests.method=testVersionsAreReturned -Dtests.seed=9919E1BF1324B5F4 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=de-DE -Dtests.timezone=America/Indiana/Petersburg -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   4.66s J2 | CloudSolrClientTest.testVersionsAreReturned <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://127.0.0.1:41935/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]    > <head>
   [junit4]    > <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
   [junit4]    > <title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
   [junit4]    > </head>
   [junit4]    > <body><h2>HTTP ERROR 404</h2>
   [junit4]    > <p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
   [junit4]    > <pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>
   [junit4]    > </body>
   [junit4]    > </html>
   [junit4]    > at __randomizedtesting.SeedInfo.seed([9919E1BF1324B5F4:61DF188A803D2D3C]:0)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
   [junit4]    > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
   [junit4]    > at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:237)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testVersionsAreReturned(CloudSolrClientTest.java:725)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:41935/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]    > <head>
   [junit4]    > <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
   [junit4]    > <title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
   [junit4]    > </head>
   [junit4]    > <body><h2>HTTP ERROR 404</h2>
   [junit4]    > <p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
   [junit4]    > <pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>
   [junit4]    > </body>
   [junit4]    > </html>
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
   [junit4]    > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
   [junit4]    > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]    > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    > ... 1 more
   [junit4]   1> 22:26:51.151 [SUITE-CloudSolrClientTest-seed#[9919E1BF1324B5F4]-worker] ERROR org.apache.zookeeper.server.ZooKeeperServer - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.impl.CloudSolrClientTest_9919E1BF1324B5F4-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {multiDefault=Lucene50(blocksize=128), a_t=Lucene50(blocksize=128), title_s=Lucene50(blocksize=128), id=FST50, text=PostingsFormat(name=Memory)}, docValues:{_version_=DocValuesFormat(name=Memory), multiDefault=DocValuesFormat(name=Lucene70), a_t=DocValuesFormat(name=Lucene70), title_s=DocValuesFormat(name=Lucene70), intDefault=DocValuesFormat(name=Memory), id=DocValuesFormat(name=Lucene70), text=DocValuesFormat(name=Direct), timestamp=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1316, maxMBSortInHeap=5.869300803513228, sim=RandomSimilarity(queryNorm=false): {}, locale=de-DE, timezone=America/Indiana/Petersburg
   [junit4]   2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=4,threads=1,free=179963528,total=438829056
   [junit4]   2> NOTE: All tests run in this JVM: [HttpSolrClientBadInputTest, SolrExampleStreamingTest, ConcurrentUpdateSolrClientMultiCollectionTest, QueryResponseTest, LessThanEvaluatorTest, NotEvaluatorTest, TestLang, JdbcDriverTest, TestSolrJErrorHandling, ClientUtilsTest, AscEvaluatorTest, EmpiricalDistributionEvaluatorTest, StreamingTest, TestDelegationTokenResponse, ArcSineEvaluatorTest, SolrExampleXMLTest, FieldValueEvaluatorTest, AndEvaluatorTest, TestPathTrie, SquareRootEvaluatorTest, LargeVolumeEmbeddedTest, CollectionAdminRequestRequiredParamsTest, StreamExpressionParserTest, MathExpressionTest, StreamExpressionTest, FacetFieldTest, CloudSolrClientTest]
   [junit4] Completed [140/162 (1!)] on J2 in 72.71s, 16 tests, 1 error <<< FAILURES!

[...truncated 7950 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/build.xml:661: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/build.xml:585: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid22393.hprof

Total time: 281 minutes 30 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-NightlyTests-7.x - Build # 34 - Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-7.x/34/

6 tests failed.
FAILED:  org.apache.solr.cloud.ShardRoutingTest.test

Error Message:
expected:<3> but was:<5>

Stack Trace:
java.lang.AssertionError: expected:<3> but was:<5>
        at __randomizedtesting.SeedInfo.seed([724D68C8926F4AC2:FA1957123C93273A]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.cloud.ShardRoutingTest.doTestNumRequests(ShardRoutingTest.java:256)
        at org.apache.solr.cloud.ShardRoutingTest.test(ShardRoutingTest.java:110)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([724D68C8926F4AC2:FA1957123C93273A]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:228)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:148)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testCollectionDoesntExist

Error Message:
Tried fetching cluster state using the node names we knew of, i.e. [127.0.0.1:43588_solr, 127.0.0.1:34904_solr, 127.0.0.1:37263_solr]. However, succeeded in obtaining the cluster state from none of them.If you think your Solr cluster is up and is accessible, you could try re-creating a new CloudSolrClient using working solrUrl(s) or zkHost(s).

Stack Trace:
java.lang.RuntimeException: Tried fetching cluster state using the node names we knew of, i.e. [127.0.0.1:43588_solr, 127.0.0.1:34904_solr, 127.0.0.1:37263_solr]. However, succeeded in obtaining the cluster state from none of them.If you think your Solr cluster is up and is accessible, you could try re-creating a new CloudSolrClient using working solrUrl(s) or zkHost(s).
        at __randomizedtesting.SeedInfo.seed([19D4027DAFC9E375:7B771CD3AD4B8765]:0)
        at org.apache.solr.client.solrj.impl.HttpClusterStateProvider.getState(HttpClusterStateProvider.java:108)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.resolveAliases(CloudSolrClient.java:1115)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:844)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:177)
        at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testCollectionDoesntExist(CloudSolrClientTest.java:779)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting

Error Message:
Error from server at http://127.0.0.1:37263/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason: <pre>    Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>  </body> </html>

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://127.0.0.1:37263/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at __randomizedtesting.SeedInfo.seed([19D4027DAFC9E375:DB633E15AC89130D]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting(CloudSolrClientTest.java:238)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:37263/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaFromClusterState

Error Message:
Collection not found: deleteFromClusterState_false

Stack Trace:
org.apache.solr.common.SolrException: Collection not found: deleteFromClusterState_false
        at __randomizedtesting.SeedInfo.seed([724D68C8926F4AC2:9CD4C3A5AD513175]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:851)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:177)
        at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
        at org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaFromClusterState(DeleteReplicaTest.java:189)
        at org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaFromClusterState(DeleteReplicaTest.java:180)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testBelowSearchRate

Error Message:
[Op{action=DELETEREPLICA, hints={COLL_SHARD=[{   "first":"belowRate_collection",   "second":"shard1"}], REPLICA=[core_node10]}}, Op{action=DELETEREPLICA, hints={COLL_SHARD=[{   "first":"belowRate_collection",   "second":"shard1"}], REPLICA=[core_node8]}}, Op{action=DELETEREPLICA, hints={COLL_SHARD=[{   "first":"belowRate_collection",   "second":"shard1"}], REPLICA=[core_node6]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:41583_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:34955_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:45514_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:44548_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:43233_solr]}}] expected:<7> but was:<8>

Stack Trace:
java.lang.AssertionError: [Op{action=DELETEREPLICA, hints={COLL_SHARD=[{
  "first":"belowRate_collection",
  "second":"shard1"}], REPLICA=[core_node10]}}, Op{action=DELETEREPLICA, hints={COLL_SHARD=[{
  "first":"belowRate_collection",
  "second":"shard1"}], REPLICA=[core_node8]}}, Op{action=DELETEREPLICA, hints={COLL_SHARD=[{
  "first":"belowRate_collection",
  "second":"shard1"}], REPLICA=[core_node6]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:41583_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:34955_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:45514_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:44548_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:43233_solr]}}] expected:<7> but was:<8>
        at __randomizedtesting.SeedInfo.seed([724D68C8926F4AC2:18E0CEA8CA87CFD1]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testBelowSearchRate(SearchRateTriggerIntegrationTest.java:400)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13809 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest
   [junit4]   2> 2275972 INFO  (SUITE-SearchRateTriggerIntegrationTest-seed#[724D68C8926F4AC2]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SearchRateTriggerIntegrationTest_724D68C8926F4AC2-001/init-core-data-001
   [junit4]   2> 2275973 WARN  (SUITE-SearchRateTriggerIntegrationTest-seed#[724D68C8926F4AC2]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=32 numCloses=32
   [junit4]   2> 2275985 INFO  (SUITE-SearchRateTriggerIntegrationTest-seed#[724D68C8926F4AC2]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2275987 INFO  (SUITE-SearchRateTriggerIntegrationTest-seed#[724D68C8926F4AC2]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2276007 INFO  (SUITE-SearchRateTriggerIntegrationTest-seed#[724D68C8926F4AC2]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 5 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SearchRateTriggerIntegrationTest_724D68C8926F4AC2-001/tempDir-001
   [junit4]   2> 2276007 INFO  (SUITE-SearchRateTriggerIntegrationTest-seed#[724D68C8926F4AC2]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2276024 INFO  (Thread-1866) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2276024 INFO  (Thread-1866) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2276070 ERROR (Thread-1866) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2276130 INFO  (SUITE-SearchRateTriggerIntegrationTest-seed#[724D68C8926F4AC2]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:44990
   [junit4]   2> 2276216 INFO  (zkConnectionManagerCallback-7461-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2276359 INFO  (jetty-launcher-7458-thread-4) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
   [junit4]   2> 2276360 INFO  (jetty-launcher-7458-thread-3) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
   [junit4]   2> 2276382 INFO  (jetty-launcher-7458-thread-1) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
   [junit4]   2> 2276399 INFO  (jetty-launcher-7458-thread-5) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
   [junit4]   2> 2276400 INFO  (jetty-launcher-7458-thread-2) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
   [junit4]   2> 2276443 INFO  (jetty-launcher-7458-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2276443 INFO  (jetty-launcher-7458-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2276443 INFO  (jetty-launcher-7458-thread-3) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2276443 INFO  (jetty-launcher-7458-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@10ad3c27{/solr,null,AVAILABLE}
   [junit4]   2> 2276444 INFO  (jetty-launcher-7458-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2276444 INFO  (jetty-launcher-7458-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2276444 INFO  (jetty-launcher-7458-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2276444 INFO  (jetty-launcher-7458-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4a41bd2a{/solr,null,AVAILABLE}
   [junit4]   2> 2276445 INFO  (jetty-launcher-7458-thread-5) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2276445 INFO  (jetty-launcher-7458-thread-5) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2276445 INFO  (jetty-launcher-7458-thread-5) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2276460 INFO  (jetty-launcher-7458-thread-5) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6c20e0ac{/solr,null,AVAILABLE}
   [junit4]   2> 2276460 INFO  (jetty-launcher-7458-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7d968b72{SSL,[ssl, http/1.1]}{127.0.0.1:45514}
   [junit4]   2> 2276460 INFO  (jetty-launcher-7458-thread-2) [    ] o.e.j.s.Server Started @2276736ms
   [junit4]   2> 2276460 INFO  (jetty-launcher-7458-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=45514}
   [junit4]   2> 2276460 ERROR (jetty-launcher-7458-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2276460 INFO  (jetty-launcher-7458-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2276460 INFO  (jetty-launcher-7458-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 2276460 INFO  (jetty-launcher-7458-thread-5) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1406a72c{SSL,[ssl, http/1.1]}{127.0.0.1:43233}
   [junit4]   2> 2276460 INFO  (jetty-launcher-7458-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2276460 INFO  (jetty-launcher-7458-thread-5) [    ] o.e.j.s.Server Started @2276736ms
   [junit4]   2> 2276460 INFO  (jetty-launcher-7458-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2276460 INFO  (jetty-launcher-7458-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-06T22:51:43.255Z
   [junit4]   2> 2276460 INFO  (jetty-launcher-7458-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43233}
   [junit4]   2> 2276461 ERROR (jetty-launcher-7458-thread-5) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2276461 INFO  (jetty-launcher-7458-thread-5) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2276461 INFO  (jetty-launcher-7458-thread-5) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 2276461 INFO  (jetty-launcher-7458-thread-5) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2276461 INFO  (jetty-launcher-7458-thread-5) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2276461 INFO  (jetty-launcher-7458-thread-5) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-06T22:51:43.256Z
   [junit4]   2> 2276461 INFO  (jetty-launcher-7458-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2b7cb11e{SSL,[ssl, http/1.1]}{127.0.0.1:41583}
   [junit4]   2> 2276461 INFO  (jetty-launcher-7458-thread-3) [    ] o.e.j.s.Server Started @2276737ms
   [junit4]   2> 2276461 INFO  (jetty-launcher-7458-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41583}
   [junit4]   2> 2276461 ERROR (jetty-launcher-7458-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2276461 INFO  (jetty-launcher-7458-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2276462 INFO  (jetty-launcher-7458-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 2276462 INFO  (jetty-launcher-7458-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2276462 INFO  (jetty-launcher-7458-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2276462 INFO  (jetty-launcher-7458-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-06T22:51:43.257Z
   [junit4]   2> 2276483 INFO  (jetty-launcher-7458-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2276483 INFO  (jetty-launcher-7458-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2276483 INFO  (jetty-launcher-7458-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2276516 INFO  (jetty-launcher-7458-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5b616ca6{/solr,null,AVAILABLE}
   [junit4]   2> 2276517 INFO  (jetty-launcher-7458-thread-4) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2276517 INFO  (jetty-launcher-7458-thread-4) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2276517 INFO  (jetty-launcher-7458-thread-4) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2276518 INFO  (jetty-launcher-7458-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4edd9244{/solr,null,AVAILABLE}
   [junit4]   2> 2276518 INFO  (jetty-launcher-7458-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@673856fd{SSL,[ssl, http/1.1]}{127.0.0.1:44548}
   [junit4]   2> 2276518 INFO  (jetty-launcher-7458-thread-4) [    ] o.e.j.s.Server Started @2276794ms
   [junit4]   2> 2276518 INFO  (jetty-launcher-7458-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44548}
   [junit4]   2> 2276518 ERROR (jetty-launcher-7458-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2276518 INFO  (jetty-launcher-7458-thread-4) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2276518 INFO  (jetty-launcher-7458-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 2276518 INFO  (jetty-launcher-7458-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2276518 INFO  (jetty-launcher-7458-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2276518 INFO  (jetty-launcher-7458-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-06T22:51:43.313Z
   [junit4]   2> 2276589 INFO  (jetty-launcher-7458-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@66ef1fb{SSL,[ssl, http/1.1]}{127.0.0.1:34955}
   [junit4]   2> 2276589 INFO  (jetty-launcher-7458-thread-1) [    ] o.e.j.s.Server Started @2276865ms
   [junit4]   2> 2276589 INFO  (jetty-launcher-7458-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34955}
   [junit4]   2> 2276590 ERROR (jetty-launcher-7458-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2276590 INFO  (jetty-launcher-7458-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2276590 INFO  (jetty-launcher-7458-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 2276590 INFO  (jetty-launcher-7458-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2276590 INFO  (jetty-launcher-7458-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2276590 INFO  (jetty-launcher-7458-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-06T22:51:43.385Z
   [junit4]   2> 2276670 INFO  (zkConnectionManagerCallback-7465-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2276671 INFO  (jetty-launcher-7458-thread-5) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2276755 INFO  (zkConnectionManagerCallback-7471-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2276756 INFO  (jetty-launcher-7458-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2276780 INFO  (zkConnectionManagerCallback-7469-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2276798 INFO  (zkConnectionManagerCallback-7467-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2276800 INFO  (zkConnectionManagerCallback-7463-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2276821 INFO  (jetty-launcher-7458-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2276821 INFO  (jetty-launcher-7458-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2276837 INFO  (jetty-launcher-7458-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2277570 INFO  (jetty-launcher-7458-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44990/solr
   [junit4]   2> 2277620 INFO  (zkConnectionManagerCallback-7475-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2277718 INFO  (jetty-launcher-7458-thread-5) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44990/solr
   [junit4]   2> 2277750 INFO  (zkConnectionManagerCallback-7477-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2277800 INFO  (jetty-launcher-7458-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44990/solr
   [junit4]   2> 2277824 INFO  (zkConnectionManagerCallback-7481-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2277836 INFO  (zkConnectionManagerCallback-7487-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2277841 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1058f7bfbf90009, likely client has closed socket
   [junit4]   2> 2277986 INFO  (zkConnectionManagerCallback-7489-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2278009 INFO  (jetty-launcher-7458-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44990/solr
   [junit4]   2> 2278061 INFO  (jetty-launcher-7458-thread-1) [n:127.0.0.1:34955_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34955_solr
   [junit4]   2> 2278063 INFO  (jetty-launcher-7458-thread-1) [n:127.0.0.1:34955_solr    ] o.a.s.c.Overseer Overseer (id=73622731592630282-127.0.0.1:34955_solr-n_0000000000) starting
   [junit4]   2> 2278072 INFO  (zkConnectionManagerCallback-7491-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2278072 INFO  (zkConnectionManagerCallback-7495-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2278089 INFO  (jetty-launcher-7458-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44990/solr
   [junit4]   2> 2278098 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1058f7bfbf9000c, likely client has closed socket
   [junit4]   2> 2278128 INFO  (zkConnectionManagerCallback-7501-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2278171 INFO  (zkConnectionManagerCallback-7503-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2278234 INFO  (zkConnectionManagerCallback-7510-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2278255 INFO  (zkConnectionManagerCallback-7512-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2278255 INFO  (jetty-launcher-7458-thread-5) [n:127.0.0.1:43233_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43233_solr
   [junit4]   2> 2278260 INFO  (zkCallback-7490-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2278260 INFO  (zkCallback-7488-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2278275 INFO  (jetty-launcher-7458-thread-1) [n:127.0.0.1:34955_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44990/solr ready
   [junit4]   2> 2278275 INFO  (zkCallback-7509-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2278275 INFO  (jetty-launcher-7458-thread-4) [n:127.0.0.1:44548_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2278277 INFO  (OverseerStateUpdate-73622731592630282-127.0.0.1:34955_solr-n_0000000000) [n:127.0.0.1:34955_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:34955_solr
   [junit4]   2> 2278287 DEBUG (OverseerAutoScalingTriggerThread-73622731592630282-127.0.0.1:34955_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 2278288 DEBUG (OverseerAutoScalingTriggerThread-73622731592630282-127.0.0.1:34955_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 2278289 DEBUG (OverseerAutoScalingTriggerThread-73622731592630282-127.0.0.1:34955_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 2278289 DEBUG (OverseerAutoScalingTriggerThread-73622731592630282-127.0.0.1:34955_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 2278290 INFO  (jetty-launcher-7458-thread-1) [n:127.0.0.1:34955_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2278290 INFO  (jetty-launcher-7458-thread-1) [n:127.0.0.1:34955_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34955_solr
   [junit4]   2> 2278331 INFO  (zkCallback-7488-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2278331 INFO  (zkCallback-7509-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2278347 INFO  (zkCallback-7502-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2278348 INFO  (zkCallback-7490-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2278349 INFO  (jetty-launcher-7458-thread-4) [n:127.0.0.1:44548_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2278349 INFO  (jetty-launcher-7458-thread-4) [n:127.0.0.1:44548_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44548_solr
   [junit4]   2> 2278350 INFO  (zkCallback-7509-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2278362 INFO  (jetty-launcher-7458-thread-2) [n:127.0.0.1:45514_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2278363 INFO  (zkCallback-7502-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2278364 INFO  (zkCallback-7490-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2278376 INFO  (zkCallback-7488-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2278376 DEBUG (OverseerAutoScalingTriggerThread-73622731592630282-127.0.0.1:34955_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:34955_solr, 127.0.0.1:44548_solr, 127.0.0.1:43233_solr]
   [junit4]   2> 2278384 DEBUG (OverseerAutoScalingTriggerThread-73622731592630282-127.0.0.1:34955_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2278387 DEBUG (OverseerAutoScalingTriggerThread-73622731592630282-127.0.0.1:34955_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 2278387 DEBUG (ScheduledTrigger-3121-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 2278387 INFO  (zkConnectionManagerCallback-7521-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2278431 INFO  (jetty-launcher-7458-thread-5) [n:127.0.0.1:43233_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2278432 INFO  (jetty-launcher-7458-thread-1) [n:127.0.0.1:34955_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2278432 INFO  (jetty-launcher-7458-thread-2) [n:127.0.0.1:45514_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2278432 INFO  (jetty-launcher-7458-thread-2) [n:127.0.0.1:45514_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45514_solr
   [junit4]   2> 2278433 INFO  (jetty-launcher-7458-thread-5) [n:127.0.0.1:43233_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44990/solr ready
   [junit4]   2> 2278434 INFO  (zkCallback-7490-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2278434 INFO  (zkCallback-7509-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2278449 INFO  (zkCallback-7502-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2278461 INFO  (jetty-launcher-7458-thread-5) [n:127.0.0.1:43233_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2278490 INFO  (zkConnectionManagerCallback-7528-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2278492 INFO  (jetty-launcher-7458-thread-4) [n:127.0.0.1:44548_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2278600 INFO  (zkConnectionManagerCallback-7533-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2278607 INFO  (jetty-launcher-7458-thread-4) [n:127.0.0.1:44548_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44990/solr ready
   [junit4]   2> 2278608 INFO  (jetty-launcher-7458-thread-4) [n:127.0.0.1:44548_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2278611 INFO  (zkCallback-7488-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2278623 INFO  (jetty-launcher-7458-thread-5) [n:127.0.0.1:43233_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43233.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b0c21b9
   [junit4]   2> 2278636 INFO  (zkCallback-7520-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2278636 INFO  (zkCallback-7511-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2278637 INFO  (jetty-launcher-7458-thread-2) [n:127.0.0.1:45514_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2278638 INFO  (jetty-launcher-7458-thread-2) [n:127.0.0.1:45514_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44990/solr ready
   [junit4]   2> 2278657 INFO  (jetty-launcher-7458-thread-1) [n:127.0.0.1:34955_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34955.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b0c21b9
   [junit4]   2> 2278664 INFO  (jetty-launcher-7458-thread-2) [n:127.0.0.1:45514_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2278676 INFO  (jetty-launcher-7458-thread-1) [n:127.0.0.1:34955_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34955.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b0c21b9
   [junit4]   2> 2278676 INFO  (jetty-launcher-7458-thread-1) [n:127.0.0.1:34955_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34955.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b0c21b9
   [junit4]   2> 2278694 INFO  (jetty-launcher-7458-thread-1) [n:127.0.0.1:34955_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SearchRateTriggerIntegrationTest_724D68C8926F4AC2-001/tempDir-001/node1/.
   [junit4]   2> 2278696 INFO  (jetty-launcher-7458-thread-4) [n:127.0.0.1:44548_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44548.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b0c21b9
   [junit4]   2> 2278696 INFO  (jetty-launcher-7458-thread-5) [n:127.0.0.1:43233_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43233.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b0c21b9
   [junit4]   2> 2278696 INFO  (jetty-launcher-7458-thread-5) [n:127.0.0.1:43233_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43233.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b0c21b9
   [junit4]   2> 2278708 INFO  (jetty-launcher-7458-thread-5) [n:127.0.0.1:43233_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SearchRateTriggerIntegrationTest_724D68C8926F4AC2-001/tempDir-001/node5/.
   [junit4]   2> 2278721 INFO  (jetty-launcher-7458-thread-2) [n:127.0.0.1:45514_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45514.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b0c21b9
   [junit4]   2> 2278731 INFO  (jetty-launcher-7458-thread-4) [n:127.0.0.1:44548_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44548.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b0c21b9
   [junit4]   2> 2278731 INFO  (jetty-launcher-7458-thread-4) [n:127.0.0.1:44548_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44548.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b0c21b9
   [junit4]   2> 2278732 INFO  (jetty-launcher-7458-thread-4) [n:127.0.0.1:44548_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SearchRateTriggerIntegrationTest_724D68C8926F4AC2-001/tempDir-001/node3/.
   [junit4]   2> 2278754 INFO  (jetty-launcher-7458-thread-3) [n:127.0.0.1:41583_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2278771 INFO  (jetty-launcher-7458-thread-2) [n:127.0.0.1:45514_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45514.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b0c21b9
   [junit4]   2> 2278772 INFO  (jetty-launcher-7458-thread-2) [n:127.0.0.1:45514_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45514.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b0c21b9
   [junit4]   2> 2278773 INFO  (jetty-launcher-7458-thread-2) [n:127.0.0.1:45514_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SearchRateTriggerIntegrationTest_724D68C8926F4AC2-001/tempDir-001/node2/.
   [junit4]   2> 2278828 INFO  (jetty-launcher-7458-thread-3) [n:127.0.0.1:41583_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2278828 INFO  (jetty-launcher-7458-thread-3) [n:127.0.0.1:41583_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41583_solr
   [junit4]   2> 2278832 INFO  (zkCallback-7490-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2278847 INFO  (zkCallback-7509-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2278867 INFO  (zkCallback-7488-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2278867 INFO  (zkCallback-7502-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2278900 INFO  (zkCallback-7511-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2278900 INFO  (zkCallback-7527-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2278904 INFO  (zkCallback-7520-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2278905 INFO  (zkCallback-7532-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2279013 INFO  (zkCallback-7476-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2279050 INFO  (zkConnectionManagerCallback-7540-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2279103 INFO  (jetty-launcher-7458-thread-3) [n:127.0.0.1:41583_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 2279104 INFO  (jetty-launcher-7458-thread-3) [n:127.0.0.1:41583_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44990/solr ready
   [junit4]   2> 2279111 INFO  (jetty-launcher-7458-thread-3) [n:127.0.0.1:41583_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2279189 INFO  (jetty-launcher-7458-thread-3) [n:127.0.0.1:41583_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41583.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b0c21b9
   [junit4]   2> 2279208 INFO  (jetty-launcher-7458-thread-3) [n:127.0.0.1:41583_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41583.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b0c21b9
   [junit4]   2> 2279208 INFO  (jetty-launcher-7458-thread-3) [n:127.0.0.1:41583_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41583.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b0c21b9
   [junit4]   2> 2279226 INFO  (jetty-launcher-7458-thread-3) [n:127.0.0.1:41583_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SearchRateTriggerIntegrationTest_724D68C8926F4AC2-001/tempDir-001/node4/.
   [junit4]   2> 2279431 DEBUG (ScheduledTrigger-3121-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2279877 INFO  (zkConnectionManagerCallback-7543-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2279880 INFO  (zkConnectionManagerCallback-7548-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2279880 INFO  (SUITE-SearchRateTriggerIntegrationTest-seed#[724D68C8926F4AC2]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 2279881 INFO  (SUITE-SearchRateTriggerIntegrationTest-seed#[724D68C8926F4AC2]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44990/solr ready
   [junit4]   2> 2280093 INFO  (qtp475033174-14780) [n:127.0.0.1:34955_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=16
   [junit4]   2> 2280253 INFO  (qtp404339128-14750) [n:127.0.0.1:41583_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2280363 INFO  (qtp290893677-14764) [n:127.0.0.1:43233_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2280440 INFO  (qtp1704047946-14786) [n:127.0.0.1:44548_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2280505 INFO  (qtp1191756972-14758) [n:127.0.0.1:45514_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2280506 DEBUG (qtp475033174-14777) [n:127.0.0.1:34955_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 2280506 INFO  (qtp475033174-14777) [n:127.0.0.1:34955_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=498
   [junit4]   2> 2280508 DEBUG (zkCallback-7509-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 2280540 DEBUG (OverseerAutoScalingTriggerThread-73622731592630282-127.0.0.1:34955_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 2280626 DEBUG (ScheduledTrigger-3121-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2280752 DEBUG (OverseerAutoScalingTriggerThread-73622731592630282-127.0.0.1:34955_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2280752 DEBUG (OverseerAutoScalingTriggerThread-73622731592630282-127.0.0.1:34955_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 2281423 INFO  (TEST-SearchRateTriggerIntegrationTest.testAboveSearchRate-seed#[724D68C8926F4AC2]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testAboveSearchRate
   [junit4]   2> 2281441 INFO  (TEST-SearchRateTriggerIntegrationTest.testAboveSearchRate-seed#[724D68C8926F4AC2]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 2281443 INFO  (TEST-SearchRateTriggerIntegrationTest.testAboveSearchRate-seed#[724D68C8926F4AC2]) [    ] o.a.s.c.a.SearchRateTriggerIntegrationTest /autoscaling.json reset, new znode version 3
   [junit4]   2> 2281486 DEBUG (zkCallback-7509-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 2281486 DEBUG (OverseerAutoScalingTriggerThread-73622731592630282-127.0.0.1:34955_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 2281487 DEBUG (OverseerAutoScalingTriggerThread-73622731592630282-127.0.0.1:34955_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 2281487 DEBUG (OverseerAutoScalingTriggerThread-73622731592630282-127.0.0.1:34955_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2281488 DEBUG (OverseerAutoScalingTriggerThread-73622731592630282-127.0.0.1:34955_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 2286446 INFO  (qtp475033174-14779) [n:127.0.0.1:34955_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf&name=aboveRate_collection&nrtReplicas=2&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2286448 INFO  (OverseerThreadFactory-3123-thread-1-processing-n:127.0.0.1:34955_solr) [n:127.0.0.1:34955_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection aboveRate_collection
   [junit4]   2> 2286553 DEBUG (OverseerThreadFactory-3123-thread-1-processing-n:127.0.0.1:34955_solr) [n:127.0.0.1:34955_solr    ] o.a.s.c.s.c.a.PolicyHelper Creating a new session
   [junit4]   2> 2286564 INFO  (qtp475033174-14778) [n:127.0.0.1:34955_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2286587 INFO  (qtp404339128-14750) [n:127.0.0.1:41583_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2286591 INFO  (qtp1704047946-14788) [n:127.0.0.1:44548_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2286620 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2286625 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2286626 DEBUG (OverseerThreadFactory-3123-thread-1-processing-n:127.0.0.1:34955_solr) [n:127.0.0.1:34955_solr    ] o.a.s.c.s.c.a.PolicyHelper New session created
   [junit4]   2> 2286627 DEBUG (OverseerThreadFactory-3123-thread-1-processing-n:127.0.0.1:34955_solr) [n:127.0.0.1:34955_solr    ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 23882114359 sessionWrapper.createTime 23882114358445075, this.sessionWrapper.createTime 23882114358445075
   [junit4]   2> 2286642 INFO  (OverseerStateUpdate-73622731592630282-127.0.0.1:34955_solr-n_0000000000) [n:127.0.0.1:34955_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"aboveRate_collection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"aboveRate_collection_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:43233/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2286656 INFO  (OverseerStateUpdate-73622731592630282-127.0.0.1:34955_solr-n_0000000000) [n:127.0.0.1:34955_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"aboveRate_collection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"aboveRate_collection_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:45514/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2286899 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr    x:aboveRate_collection_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=aboveRate_collection_shard1_replica_n2&action=CREATE&numShards=1&collection=aboveRate_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2286950 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr    x:aboveRate_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=aboveRate_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=aboveRate_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2286950 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr    x:aboveRate_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2287974 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 2288035 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.s.IndexSchema [aboveRate_collection_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 2288045 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 2288046 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2288046 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'aboveRate_collection_shard1_replica_n2' using configuration from collection aboveRate_collection, trusted=true
   [junit4]   2> 2288047 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45514.solr.core.aboveRate_collection.shard1.replica_n2' (registry 'solr.core.aboveRate_collection.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b0c21b9
   [junit4]   2> 2288047 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.c.SolrCore [[aboveRate_collection_shard1_replica_n2] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SearchRateTriggerIntegrationTest_724D68C8926F4AC2-001/tempDir-001/node2/aboveRate_collection_shard1_replica_n2], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SearchRateTriggerIntegrationTest_724D68C8926F4AC2-001/tempDir-001/node2/./aboveRate_collection_shard1_replica_n2/data/]
   [junit4]   2> 2288068 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.s.IndexSchema [aboveRate_collection_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 2288070 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2288070 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'aboveRate_collection_shard1_replica_n1' using configuration from collection aboveRate_collection, trusted=true
   [junit4]   2> 2288071 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43233.solr.core.aboveRate_collection.shard1.replica_n1' (registry 'solr.core.aboveRate_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b0c21b9
   [junit4]   2> 2288071 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.c.SolrCore [[aboveRate_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SearchRateTriggerIntegrationTest_724D68C8926F4AC2-001/tempDir-001/node5/aboveRate_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SearchRateTriggerIntegrationTest_724D68C8926F4AC2-001/tempDir-001/node5/./aboveRate_collection_shard1_replica_n1/data/]
   [junit4]   2> 2288273 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2288273 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2288274 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2288274 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2288277 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2288277 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2288279 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2288279 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2288316 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@31aee724[aboveRate_collection_shard1_replica_n2] main]
   [junit4]   2> 2288317 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2288317 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2288318 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2288318 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1616426791194329088
   [junit4]   2> 2288318 INFO  (searcherExecutor-3144-thread-1-processing-n:127.0.0.1:45514_solr x:aboveRate_collection_shard1_replica_n2 c:aboveRate_collection s:shard1 r:core_node4) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.c.SolrCore [aboveRate_collection_shard1_replica_n2] Registered new searcher Searcher@31aee724[aboveRate_collection_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2288335 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/aboveRate_collection/terms/shard1 to Terms{values={core_node4=0}, version=0}
   [junit4]   2> 2288356 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7583d0bd[aboveRate_collection_shard1_replica_n1] main]
   [junit4]   2> 2288368 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9979ms
   [junit4]   2> 2288369 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2288369 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2288370 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2288370 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1616426791248855040
   [junit4]   2> 2288371 INFO  (searcherExecutor-3145-thread-1-processing-n:127.0.0.1:43233_solr x:aboveRate_collection_shard1_replica_n1 c:aboveRate_collection s:shard1 r:core_node3) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.c.SolrCore [aboveRate_collection_shard1_replica_n1] Registered new searcher Searcher@7583d0bd[aboveRate_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2288375 INFO  (qtp290893677-14767) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/aboveRate_collection/terms/shard1 to Terms{values={core_node3=0, core_node4=0}, version=1}
   [junit4]   2> 2288870 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2288870 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2288870 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:45514/solr/aboveRate_collection_shard1_replica_n2/
   [junit4]   2> 2288870 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=aboveRate_collection_shard1_replica_n2 url=https://127.0.0.1:45514/solr START replicas=[https://127.0.0.1:43233/solr/aboveRate_collection_shard1_replica_n1/] nUpdates=100
   [junit4]   2> 2288871 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=aboveRate_collection_shard1_replica_n2 url=https://127.0.0.1:45514/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 2288877 INFO  (qtp290893677-14762) [n:127.0.0.1:43233_solr c:aboveRate_collection s:shard1 r:core_node3 x:aboveRate_collection_shard1_replica_n1] o.a.s.c.S.Request [aboveRate_collection_shard1_replica_n1]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 2288877 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 2288877 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 2288877 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2288889 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:45514/solr/aboveRate_collection_shard1_replica_n2/ shard1
   [junit4]   2> 2289080 INFO  (qtp1191756972-14754) [n:127.0.0.1:45514_solr c:aboveRate_collection s:shard1 r:core_node4 x:aboveRate_collection_shard1_re

[...truncated too long message...]

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/null903629245
     [copy] Copying 241 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/null903629245
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/null903629245

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 35ms :: artifacts dl 8ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 579 minutes 34 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-NightlyTests-7.x - Build # 35 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-7.x/35/

73 tests failed.
FAILED:  org.apache.solr.OutputWriterTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.TestDocumentBuilder.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.TestJoin.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.client.solrj.embedded.TestJettySolrRunner.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.AddReplicaTest

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.BasicZkTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.CleanupOldIndexTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.lang.reflect.Method.getTypeParameters(Method.java:216)
        at java.lang.reflect.Executable.sharedToGenericString(Executable.java:148)
        at java.lang.reflect.Method.toGenericString(Method.java:415)
        at com.carrotsearch.randomizedtesting.ClassModel$1.compare(ClassModel.java:27)
        at com.carrotsearch.randomizedtesting.ClassModel$1.compare(ClassModel.java:23)
        at java.util.TimSort.binarySort(TimSort.java:296)
        at java.util.TimSort.sort(TimSort.java:221)
        at java.util.Arrays.sort(Arrays.java:1438)
        at com.carrotsearch.randomizedtesting.ClassModel$3.members(ClassModel.java:216)
        at com.carrotsearch.randomizedtesting.ClassModel$3.members(ClassModel.java:212)
        at com.carrotsearch.randomizedtesting.ClassModel$ModelBuilder.build(ClassModel.java:85)
        at com.carrotsearch.randomizedtesting.ClassModel.methodsModel(ClassModel.java:224)
        at com.carrotsearch.randomizedtesting.ClassModel.<init>(ClassModel.java:207)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.<init>(RandomizedRunner.java:323)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:31)
        at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:24)
        at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
        at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
        at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
        at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:258)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:394)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)


FAILED:  org.apache.solr.cloud.DeleteInactiveReplicaTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.DistributedQueueTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.MigrateRouteKeyTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.OverseerRolesTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.ReplaceNodeTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.SliceStateTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.SolrXmlInZkTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.TestAuthenticationFramework.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.TestCloudPseudoReturnFields.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.TestDistribDocBasedVersion.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.TestExclusionRuleCollectionAccess.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.TestLeaderElectionZkExpiry.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.TestPullReplica.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.ZkNodePropsTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.api.collections.AssignTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMixedBounds

Error Message:
did not finish processing in time

Stack Trace:
java.lang.AssertionError: did not finish processing in time
        at __randomizedtesting.SeedInfo.seed([31F7FD76DCA1D669:3B7442DB911ADD33]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMixedBounds(IndexSizeTriggerTest.java:576)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.MetricTriggerIntegrationTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup

Error Message:
java.util.concurrent.ExecutionException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: last ClusterState: znodeVersion: 12 live nodes:[127.0.0.1:10011_solr] collections:{.system=DocCollection(.system//clusterstate.json/11)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{"shard1":{       "replicas":{"core_node1":{           "core":".system_shard1_replica_n1",           "leader":"true",           "SEARCHER.searcher.maxDoc":14,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":17240,           "node_name":"127.0.0.1:10011_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.605600118637085E-5,           "SEARCHER.searcher.numDocs":14}},       "range":"80000000-7fffffff",       "state":"active"}}}, ScheduledMaintenanceTriggerTest_collection1=DocCollection(ScheduledMaintenanceTriggerTest_collection1//clusterstate.json/11)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{     "shard1":{       "replicas":{"core_node1":{           "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_replica_n1",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10011_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"80000000-7fffffff",       "state":"active",       "__buffered_updates__":0},     "shard1_1":{       "parent":"shard1",       "stateTimestamp":"1542400166206590750",       "range":"0-7fffffff",       "state":"construction",       "replicas":{"core_node7":{           "leader":"true",           "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_1_replica_n6",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10011_solr",           "base_url":"http://127.0.0.1:10011/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}}},     "shard1_0":{       "parent":"shard1",       "stateTimestamp":"1542400166206439100",       "range":"80000000-ffffffff",       "state":"construction",       "replicas":{"core_node5":{           "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_0_replica_n4",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":20480,           "node_name":"127.0.0.1:10011_solr",           "base_url":"http://127.0.0.1:10011/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.9073486328125E-5,           "SEARCHER.searcher.numDocs":0}}}}}}, last coll state: DocCollection(ScheduledMaintenanceTriggerTest_collection1//clusterstate.json/11)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{     "shard1":{       "replicas":{"core_node1":{           "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_replica_n1",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10011_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"80000000-7fffffff",       "state":"active",       "__buffered_updates__":0},     "shard1_1":{       "parent":"shard1",       "stateTimestamp":"1542400166206590750",       "range":"0-7fffffff",       "state":"construction",       "replicas":{"core_node7":{           "leader":"true",           "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_1_replica_n6",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10011_solr",           "base_url":"http://127.0.0.1:10011/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}}},     "shard1_0":{       "parent":"shard1",       "stateTimestamp":"1542400166206439100",       "range":"80000000-ffffffff",       "state":"construction",       "replicas":{"core_node5":{           "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_0_replica_n4",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":20480,           "node_name":"127.0.0.1:10011_solr",           "base_url":"http://127.0.0.1:10011/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.9073486328125E-5,           "SEARCHER.searcher.numDocs":0}}}}}

Stack Trace:
java.io.IOException: java.util.concurrent.ExecutionException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: last ClusterState: znodeVersion: 12
live nodes:[127.0.0.1:10011_solr]
collections:{.system=DocCollection(.system//clusterstate.json/11)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{"shard1":{
      "replicas":{"core_node1":{
          "core":".system_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10011_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14}},
      "range":"80000000-7fffffff",
      "state":"active"}}}, ScheduledMaintenanceTriggerTest_collection1=DocCollection(ScheduledMaintenanceTriggerTest_collection1//clusterstate.json/11)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard1":{
      "replicas":{"core_node1":{
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10011_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active",
      "__buffered_updates__":0},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1542400166206590750",
      "range":"0-7fffffff",
      "state":"construction",
      "replicas":{"core_node7":{
          "leader":"true",
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_1_replica_n6",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10011_solr",
          "base_url":"http://127.0.0.1:10011/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1542400166206439100",
      "range":"80000000-ffffffff",
      "state":"construction",
      "replicas":{"core_node5":{
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_0_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":20480,
          "node_name":"127.0.0.1:10011_solr",
          "base_url":"http://127.0.0.1:10011/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.9073486328125E-5,
          "SEARCHER.searcher.numDocs":0}}}}}}, last coll state: DocCollection(ScheduledMaintenanceTriggerTest_collection1//clusterstate.json/11)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard1":{
      "replicas":{"core_node1":{
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10011_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active",
      "__buffered_updates__":0},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1542400166206590750",
      "range":"0-7fffffff",
      "state":"construction",
      "replicas":{"core_node7":{
          "leader":"true",
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_1_replica_n6",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10011_solr",
          "base_url":"http://127.0.0.1:10011/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1542400166206439100",
      "range":"80000000-ffffffff",
      "state":"construction",
      "replicas":{"core_node5":{
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_0_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":20480,
          "node_name":"127.0.0.1:10011_solr",
          "base_url":"http://127.0.0.1:10011/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.9073486328125E-5,
          "SEARCHER.searcher.numDocs":0}}}}}
        at __randomizedtesting.SeedInfo.seed([31F7FD76DCA1D669:2CDB3D04BDE2F162]:0)
        at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:654)
        at org.apache.solr.cloud.autoscaling.sim.SimCloudManager$1.request(SimCloudManager.java:224)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup(ScheduledMaintenanceTriggerTest.java:221)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: last ClusterState: znodeVersion: 12
live nodes:[127.0.0.1:10011_solr]
collections:{.system=DocCollection(.system//clusterstate.json/11)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{"shard1":{
      "replicas":{"core_node1":{
          "core":".system_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10011_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14}},
      "range":"80000000-7fffffff",
      "state":"active"}}}, ScheduledMaintenanceTriggerTest_collection1=DocCollection(ScheduledMaintenanceTriggerTest_collection1//clusterstate.json/11)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard1":{
      "replicas":{"core_node1":{
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10011_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active",
      "__buffered_updates__":0},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1542400166206590750",
      "range":"0-7fffffff",
      "state":"construction",
      "replicas":{"core_node7":{
          "leader":"true",
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_1_replica_n6",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10011_solr",
          "base_url":"http://127.0.0.1:10011/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1542400166206439100",
      "range":"80000000-ffffffff",
      "state":"construction",
      "replicas":{"core_node5":{
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_0_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":20480,
          "node_name":"127.0.0.1:10011_solr",
          "base_url":"http://127.0.0.1:10011/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.9073486328125E-5,
          "SEARCHER.searcher.numDocs":0}}}}}}, last coll state: DocCollection(ScheduledMaintenanceTriggerTest_collection1//clusterstate.json/11)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard1":{
      "replicas":{"core_node1":{
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10011_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active",
      "__buffered_updates__":0},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1542400166206590750",
      "range":"0-7fffffff",
      "state":"construction",
      "replicas":{"core_node7":{
          "leader":"true",
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_1_replica_n6",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10011_solr",
          "base_url":"http://127.0.0.1:10011/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1542400166206439100",
      "range":"80000000-ffffffff",
      "state":"construction",
      "replicas":{"core_node5":{
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_0_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":20480,
          "node_name":"127.0.0.1:10011_solr",
          "base_url":"http://127.0.0.1:10011/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.9073486328125E-5,
          "SEARCHER.searcher.numDocs":0}}}}}
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:652)
        ... 43 more
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: last ClusterState: znodeVersion: 12
live nodes:[127.0.0.1:10011_solr]
collections:{.system=DocCollection(.system//clusterstate.json/11)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{"shard1":{
      "replicas":{"core_node1":{
          "core":".system_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10011_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14}},
      "range":"80000000-7fffffff",
      "state":"active"}}}, ScheduledMaintenanceTriggerTest_collection1=DocCollection(ScheduledMaintenanceTriggerTest_collection1//clusterstate.json/11)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard1":{
      "replicas":{"core_node1":{
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10011_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active",
      "__buffered_updates__":0},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1542400166206590750",
      "range":"0-7fffffff",
      "state":"construction",
      "replicas":{"core_node7":{
          "leader":"true",
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_1_replica_n6",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10011_solr",
          "base_url":"http://127.0.0.1:10011/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1542400166206439100",
      "range":"80000000-ffffffff",
      "state":"construction",
      "replicas":{"core_node5":{
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_0_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":20480,
          "node_name":"127.0.0.1:10011_solr",
          "base_url":"http://127.0.0.1:10011/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.9073486328125E-5,
          "SEARCHER.searcher.numDocs":0}}}}}}, last coll state: DocCollection(ScheduledMaintenanceTriggerTest_collection1//clusterstate.json/11)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard1":{
      "replicas":{"core_node1":{
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10011_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active",
      "__buffered_updates__":0},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1542400166206590750",
      "range":"0-7fffffff",
      "state":"construction",
      "replicas":{"core_node7":{
          "leader":"true",
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_1_replica_n6",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10011_solr",
          "base_url":"http://127.0.0.1:10011/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1542400166206439100",
      "range":"80000000-ffffffff",
      "state":"construction",
      "replicas":{"core_node5":{
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_0_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":20480,
          "node_name":"127.0.0.1:10011_solr",
          "base_url":"http://127.0.0.1:10011/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.9073486328125E-5,
          "SEARCHER.searcher.numDocs":0}}}}}
        at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:854)
        at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.lambda$request$15(SimCloudManager.java:651)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more
Caused by: java.util.concurrent.TimeoutException: last ClusterState: znodeVersion: 12
live nodes:[127.0.0.1:10011_solr]
collections:{.system=DocCollection(.system//clusterstate.json/11)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{"shard1":{
      "replicas":{"core_node1":{
          "core":".system_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10011_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14}},
      "range":"80000000-7fffffff",
      "state":"active"}}}, ScheduledMaintenanceTriggerTest_collection1=DocCollection(ScheduledMaintenanceTriggerTest_collection1//clusterstate.json/11)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard1":{
      "replicas":{"core_node1":{
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10011_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active",
      "__buffered_updates__":0},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1542400166206590750",
      "range":"0-7fffffff",
      "state":"construction",
      "replicas":{"core_node7":{
          "leader":"true",
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_1_replica_n6",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10011_solr",
          "base_url":"http://127.0.0.1:10011/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1542400166206439100",
      "range":"80000000-ffffffff",
      "state":"construction",
      "replicas":{"core_node5":{
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_0_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":20480,
          "node_name":"127.0.0.1:10011_solr",
          "base_url":"http://127.0.0.1:10011/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.9073486328125E-5,
          "SEARCHER.searcher.numDocs":0}}}}}}, last coll state: DocCollection(ScheduledMaintenanceTriggerTest_collection1//clusterstate.json/11)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard1":{
      "replicas":{"core_node1":{
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10011_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active",
      "__buffered_updates__":0},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1542400166206590750",
      "range":"0-7fffffff",
      "state":"construction",
      "replicas":{"core_node7":{
          "leader":"true",
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_1_replica_n6",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10011_solr",
          "base_url":"http://127.0.0.1:10011/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1542400166206439100",
      "range":"80000000-ffffffff",
      "state":"construction",
      "replicas":{"core_node5":{
          "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_0_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":20480,
          "node_name":"127.0.0.1:10011_solr",
          "base_url":"http://127.0.0.1:10011/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.9073486328125E-5,
          "SEARCHER.searcher.numDocs":0}}}}}
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:112)
        at org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.simSplitShard(SimClusterStateProvider.java:1246)
        at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:852)
        ... 6 more


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [NRTCachingDirectory] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:358)  at org.apache.solr.core.SolrCore.cleanupOldIndexDirectories(SolrCore.java:3093)  at org.apache.solr.core.SolrCore.close(SolrCore.java:1614)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1006)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:869)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1138)  at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:684)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [NRTCachingDirectory]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:358)
        at org.apache.solr.core.SolrCore.cleanupOldIndexDirectories(SolrCore.java:3093)
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1614)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1006)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:869)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1138)
        at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:684)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


        at __randomizedtesting.SeedInfo.seed([31F7FD76DCA1D669]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:304)
        at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsRecoverLeaseTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.overseer.TestClusterStateMutator.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.rule.ImplicitSnitchTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.core.HdfsDirectoryFactoryTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.core.TestConfigSetImmutable.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.core.TestCoreContainer.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.core.TestMergePolicyConfig.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.core.TestXIncludeConfig.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.handler.admin.SecurityConfHandlerTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.SuggestComponentTest

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.index.UninvertDocValuesMergePolicyTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.internal.csv.CSVParserTest.testDelimiterIsWhitespace

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.internal.csv.CSVUtilsTest.testParse4

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.metrics.reporters.SolrJmxReporterCloudTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.metrics.reporters.solr.SolrCloudReportersTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.response.transform.TestChildDocTransformerHierarchy.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.schema.RequiredFieldsTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.schema.TestCloudSchemaless.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.search.SpatialFilterTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.search.TestInitQParser.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.search.TestLegacyNumericRangeQueryBuilder.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.search.TestPayloadCheckQParserPlugin.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.search.facet.TestJsonFacetRefinement.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.search.facet.TestJsonFacetsWithNestedObjects.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.search.function.distance.DistanceFunctionTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.search.join.TestCloudNestedDocsSort.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.search.similarities.TestDFISimilarityFactory.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.search.similarities.TestLMDirichletSimilarityFactory.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.store.blockcache.BlockCacheTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at __randomizedtesting.SeedInfo.seed([31F7FD76DCA1D669:245D5B2D1160CDE]:0)
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
        at java.lang.StringBuilder.append(StringBuilder.java:136)
        at java.lang.StringBuilder.append(StringBuilder.java:131)
        at org.apache.lucene.index.IndexWriterConfig.toString(IndexWriterConfig.java:480)
        at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1163)
        at org.apache.lucene.index.RandomIndexWriter.maybeFlushOrCommit(RandomIndexWriter.java:198)
        at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:192)
        at org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit(TestDocTermOrdsUninvertLimit.java:66)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)


FAILED:  junit.framework.TestSuite.org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit

Error Message:
Captured an uncaught exception in thread: Thread[id=6420, name=Lucene Merge Thread #79, state=RUNNABLE, group=TGRP-TestDocTermOrdsUninvertLimit]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6420, name=Lucene Merge Thread #79, state=RUNNABLE, group=TGRP-TestDocTermOrdsUninvertLimit]
Caused by: org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
        at __randomizedtesting.SeedInfo.seed([31F7FD76DCA1D669]:0)
        at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:704)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684)
Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
        at org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349)
        at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669)
        at org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664)
        at org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5017)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4532)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068)
        at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:84)
        at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:57)
        at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:168)
        at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:154)
        at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:141)
        at org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:73)
        at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.addPosition(Lucene50PostingsWriter.java:291)
        at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:156)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
        at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
        at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
        at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:231)
        at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4446)
        ... 3 more


FAILED:  junit.framework.TestSuite.org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit

Error Message:
Captured an uncaught exception in thread: Thread[id=6427, name=Lucene Merge Thread #86, state=RUNNABLE, group=TGRP-TestDocTermOrdsUninvertLimit]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6427, name=Lucene Merge Thread #86, state=RUNNABLE, group=TGRP-TestDocTermOrdsUninvertLimit]
Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
        at __randomizedtesting.SeedInfo.seed([31F7FD76DCA1D669]:0)
        at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:704)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:84)
        at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:57)
        at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:168)
        at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:154)
        at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:141)
        at org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:73)
        at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.addPosition(Lucene50PostingsWriter.java:291)
        at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:156)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
        at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
        at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
        at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:231)
        at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4446)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068)
        at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)


FAILED:  org.apache.solr.uninverting.TestFieldCacheSort.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.update.SolrIndexSplitterTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.update.TestUpdate.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.update.processor.ScriptEngineTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.update.processor.UpdateRequestProcessorFactoryTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.util.NumberUtilsTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.util.TestMaxTokenLenTokenizer.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.util.TestSafeXMLParsing.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded




Build Log:
[...truncated 13851 lines...]
   [junit4] Suite: org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDocTermOrdsUninvertLimit -Dtests.method=testTriggerUnInvertLimit -Dtests.seed=31F7FD76DCA1D669 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=mk -Dtests.timezone=America/Goose_Bay -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   83.1s J2 | TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at __randomizedtesting.SeedInfo.seed([31F7FD76DCA1D669:245D5B2D1160CDE]:0)
   [junit4]    > at java.util.Arrays.copyOf(Arrays.java:3332)
   [junit4]    > at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
   [junit4]    > at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
   [junit4]    > at java.lang.StringBuilder.append(StringBuilder.java:136)
   [junit4]    > at java.lang.StringBuilder.append(StringBuilder.java:131)
   [junit4]    > at org.apache.lucene.index.IndexWriterConfig.toString(IndexWriterConfig.java:480)
   [junit4]    > at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1163)
   [junit4]    > at org.apache.lucene.index.RandomIndexWriter.maybeFlushOrCommit(RandomIndexWriter.java:198)
   [junit4]    > at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:192)
   [junit4]    > at org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit(TestDocTermOrdsUninvertLimit.java:66)
   [junit4]   2> Nov 14, 2018 11:57:04 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Lucene Merge Thread #79,5,TGRP-TestDocTermOrdsUninvertLimit]
   [junit4]   2> org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
   [junit4]   2> at __randomizedtesting.SeedInfo.seed([31F7FD76DCA1D669]:0)
   [junit4]   2> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:704)
   [junit4]   2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684)
   [junit4]   2> Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
   [junit4]   2> at org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349)
   [junit4]   2> at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669)
   [junit4]   2> at org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5017)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4532)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068)
   [junit4]   2> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
   [junit4]   2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:84)
   [junit4]   2> at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:57)
   [junit4]   2> at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:168)
   [junit4]   2> at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:154)
   [junit4]   2> at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:141)
   [junit4]   2> at org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:73)
   [junit4]   2> at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.addPosition(Lucene50PostingsWriter.java:291)
   [junit4]   2> at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:156)
   [junit4]   2> at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
   [junit4]   2> at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
   [junit4]   2> at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   [junit4]   2> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
   [junit4]   2> at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:231)
   [junit4]   2> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4446)
   [junit4]   2> ... 3 more
   [junit4]   2>
   [junit4]   2> Nov 14, 2018 11:57:08 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Lucene Merge Thread #86,5,TGRP-TestDocTermOrdsUninvertLimit]
   [junit4]   2> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> at __randomizedtesting.SeedInfo.seed([31F7FD76DCA1D669]:0)
   [junit4]   2> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:704)
   [junit4]   2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684)
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:84)
   [junit4]   2> at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:57)
   [junit4]   2> at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:168)
   [junit4]   2> at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:154)
   [junit4]   2> at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:141)
   [junit4]   2> at org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:73)
   [junit4]   2> at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.addPosition(Lucene50PostingsWriter.java:291)
   [junit4]   2> at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:156)
   [junit4]   2> at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
   [junit4]   2> at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
   [junit4]   2> at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   [junit4]   2> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
   [junit4]   2> at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:231)
   [junit4]   2> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4446)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068)
   [junit4]   2> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
   [junit4]   2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
   [junit4]   2>
   [junit4]   2> NOTE: test params are: codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=10, maxDocsPerChunk=2, blockSize=5), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=10, blockSize=5)), sim=RandomSimilarity(queryNorm=true): {field=DFR I(ne)L3(800.0)}, locale=mk, timezone=America/Goose_Bay
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=59543944,total=477626368
   [junit4]   2> NOTE: All tests run in this JVM: [TestSmileRequest, TestSolrQueryParser, TestTrie, TestTolerantUpdateProcessorCloud, TestPerFieldSimilarity, SpellingQueryConverterTest, DeleteReplicaTest, ConvertedLegacyTest, TestEmbeddedSolrServerAdminHandler, TestLegacyNumericUtils, RegexBytesRefFilterTest, TestFieldResource, SearchRateTriggerTest, TestComplexPhraseLeadingWildcard, CoreSorterTest, TestSolrCloudWithHadoopAuthPlugin, TestTestInjection, TestSSLRandomization, DocValuesMultiTest, TestGraphTermsQParserPlugin, TestClusterProperties, BinaryUpdateRequestHandlerTest, DistributedFacetExistsSmallTest, TestCodecSupport, ForceLeaderTest, DistribCursorPagingTest, TestQueryTypes, TestWordDelimiterFilterFactory, SolrMetricsIntegrationTest, ScheduledTriggerIntegrationTest, XsltUpdateRequestHandlerTest, DistributedQueryElevationComponentTest, CustomCollectionTest, SearchHandlerTest, PreAnalyzedFieldManagedSchemaCloudTest, NumericFieldsTest, DistributedMLTComponentTest, ConnectionManagerTest, QueryEqualityTest, AtomicUpdateProcessorFactoryTest, URLClassifyProcessorTest, TriLevelCompositeIdRoutingTest, TestSubQueryTransformerDistrib, TestDocTermOrdsUninvertLimit]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDocTermOrdsUninvertLimit -Dtests.seed=31F7FD76DCA1D669 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=mk -Dtests.timezone=America/Goose_Bay -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J2 | TestDocTermOrdsUninvertLimit (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6420, name=Lucene Merge Thread #79, state=RUNNABLE, group=TGRP-TestDocTermOrdsUninvertLimit]
   [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
   [junit4]    > at __randomizedtesting.SeedInfo.seed([31F7FD76DCA1D669]:0)
   [junit4]    > at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:704)
   [junit4]    > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684)
   [junit4]    > Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
   [junit4]    > at org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349)
   [junit4]    > at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669)
   [junit4]    > at org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664)
   [junit4]    > at org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5017)
   [junit4]    > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4532)
   [junit4]    > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068)
   [junit4]    > at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
   [junit4]    > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:84)
   [junit4]    > at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:57)
   [junit4]    > at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:168)
   [junit4]    > at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:154)
   [junit4]    > at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:141)
   [junit4]    > at org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:73)
   [junit4]    > at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.addPosition(Lucene50PostingsWriter.java:291)
   [junit4]    > at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:156)
   [junit4]    > at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
   [junit4]    > at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
   [junit4]    > at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   [junit4]    > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
   [junit4]    > at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:231)
   [junit4]    > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
   [junit4]    > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4446)
   [junit4]    > ... 3 moreThrowable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6427, name=Lucene Merge Thread #86, state=RUNNABLE, group=TGRP-TestDocTermOrdsUninvertLimit]
   [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at __randomizedtesting.SeedInfo.seed([31F7FD76DCA1D669]:0)
   [junit4]    > at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:704)
   [junit4]    > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:84)
   [junit4]    > at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:57)
   [junit4]    > at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:168)
   [junit4]    > at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:154)
   [junit4]    > at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:141)
   [junit4]    > at org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:73)
   [junit4]    > at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.addPosition(Lucene50PostingsWriter.java:291)
   [junit4]    > at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:156)
   [junit4]    > at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
   [junit4]    > at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
   [junit4]    > at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   [junit4]    > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
   [junit4]    > at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:231)
   [junit4]    > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
   [junit4]    > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4446)
   [junit4]    > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068)
   [junit4]    > at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
   [junit4]    > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
   [junit4] Completed [126/836 (1!)] on J2 in 96.84s, 1 test, 3 errors <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.cloud.CleanupOldIndexTest
   [junit4] ERROR   3.04s J2 | CleanupOldIndexTest.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > at java.lang.reflect.Method.getTypeParameters(Method.java:216)
   [junit4]    > at java.lang.reflect.Executable.sharedToGenericString(Executable.java:148)
   [junit4]    > at java.lang.reflect.Method.toGenericString(Method.java:415)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$1.compare(ClassModel.java:27)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$1.compare(ClassModel.java:23)
   [junit4]    > at java.util.TimSort.binarySort(TimSort.java:296)
   [junit4]    > at java.util.TimSort.sort(TimSort.java:221)
   [junit4]    > at java.util.Arrays.sort(Arrays.java:1438)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$3.members(ClassModel.java:216)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$3.members(ClassModel.java:212)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$ModelBuilder.build(ClassModel.java:85)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel.methodsModel(ClassModel.java:224)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel.<init>(ClassModel.java:207)
   [junit4]    > at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
   [junit4] Completed [127/836 (2!)] on J2 in 3.04s, 1 test, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestMiniSolrCloudClusterSSL
   [junit4] ERROR   0.00s J2 | TestMiniSolrCloudClusterSSL.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [128/836 (3!)] on J2 in 0.00s, 1 test, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.search.join.TestCloudNestedDocsSort
   [junit4] ERROR   0.00s J2 | TestCloudNestedDocsSort.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [129/836 (4!)] on J2 in 0.00s, 1 test, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.internal.csv.CSVUtilsTest
   [junit4] ERROR   0.93s J2 | CSVUtilsTest.testParse4 <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [130/836 (5!)] on J2 in 6.71s, 14 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.update.processor.UpdateRequestProcessorFactoryTest
   [junit4] ERROR   0.00s J2 | UpdateRequestProcessorFactoryTest.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [131/836 (6!)] on J2 in 0.00s, 1 test, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicZkTest
   [junit4] ERROR   0.00s J2 | BasicZkTest.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [132/836 (7!)] on J2 in 0.00s, 1 test, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.TestJoin
   [junit4] ERROR   0.00s J2 | TestJoin.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [133/836 (8!)] on J2 in 0.00s, 1 test, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.core.TestConfigSetImmutable
   [junit4] ERROR   0.00s J2 | TestConfigSetImmutable.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [134/836 (9!)] on J2 in 0.00s, 1 test, 1 error <<< FAILURES!

[...truncated 2 lines...]
   [junit4] Suite: org.apache.solr.OutputWriterTest
   [junit4] ERROR   0.00s J2 | OutputWriterTest.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [135/836 (10!)] on J2 in 0.00s, 1 test, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.cloud.rule.ImplicitSnitchTest
   [junit4] ERROR   0.00s J2 | ImplicitSnitchTest.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [136/836 (11!)] on J2 in 0.00s, 1 test, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.core.HdfsDirectoryFactoryTest
   [junit4] ERROR   0.00s J2 | HdfsDirectoryFactoryTest.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [137/836 (12!)] on J2 in 0.01s, 1 test, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.store.blockcache.BlockCacheTest
   [junit4] ERROR   0.00s J2 | BlockCacheTest.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [138/836 (13!)] on J2 in 0.00s, 1 test, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.search.SpatialFilterTest
   [junit4] ERROR   0.00s J2 | SpatialFilterTest.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [139/836 (14!)] on J2 in 0.00s, 1 test, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.cloud.DistributedQueueTest
   [junit4] ERROR   0.00s J2 | DistributedQueueTest.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [140/836 (15!)] on J2 in 0.00s, 1 test, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.response.transform.TestChildDocTransformerHierarchy
   [junit4] ERROR   0.00s J2 | TestChildDocTransformerHierarchy.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [141/836 (16!)] on J2 in 0.00s, 1 test, 1 error <<< FAILURES!

[...truncated 2 lines...]
   [junit4] Suite: org.apache.solr.search.facet.TestJsonFacetsWithNestedObjects
   [junit4] ERROR   0.00s J2 | TestJsonFacetsWithNestedObjects.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [142/836 (17!)] on J2 in 0.00s, 1 test, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.metrics.reporters.solr.SolrCloudReportersTest
   [junit4] ERROR   0.00s J2 | SolrCloudReportersTest.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [143/836 (18!)] on J2 in 0.00s, 1 test, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestExclusionRuleCollectionAccess
   [junit4] ERROR   0.00s J2 | TestExclusionRuleCollectionAccess.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [144/836 (19!)] on J2 in 0.00s, 1 test, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.core.TestMergePolicyConfig
   [junit4] ERROR   0.00s J2 | TestMergePolicyConfig.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [145/836 (20!)] on J2 in 0.00s, 1 test, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.util.TestMaxTokenLenTokenizer
   [junit4] ERROR   0.00s J2 | TestMaxTokenLenTokenizer.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [146/836 (21!)] on J2 in 0.00s, 1 test, 1 error <<<

[...truncated too long message...]

lr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/api-all-1.0.0-M20.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/bcprov-jdk15on-1.60.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/byte-buddy-1.6.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/ehcache-core-2.4.4.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/hadoop-common-2.7.4-tests.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/hadoop-hdfs-2.7.4-tests.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/hadoop-minikdc-2.7.4.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/jersey-core-1.9.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/jersey-server-1.9.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/jetty-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/jetty-sslengine-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/jetty-util-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/mina-core-2.0.0-M5.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/mockito-core-2.6.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/netty-all-4.0.36.Final.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/objenesis-2.5.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/build/analysis/icu/lucene-analyzers-icu-7.7.0-SNAPSHOT.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/contrib/solr-analysis-extras/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/build/queryparser/classes/test:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/build/backward-codecs/classes/test:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/contrib/analysis-extras/lib/icu4j-62.1.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/usr/local/asfpackages/java/jdk1.8.0_191/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.6.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J2-20181114_113703_398658329052423834733.events @/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J2-20181114_113703_3984482892807799593135.suites -stdin
   [junit4] ERROR: JVM J2 ended with an exception: Forked JVM ran out of memory. See process stdout at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J2-20181114_113703_3986370795574671270387.sysout See process stderr at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J2-20181114_113703_3986118956146383826178.syserr
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1542)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:123)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:997)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:994)
   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4] at java.lang.Thread.run(Thread.java:748)

BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/build.xml:661: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/build.xml:577: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/build.xml:59: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build.xml:267: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/common-build.xml:558: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/common-build.xml:1567: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/common-build.xml:1092: At least one slave process threw an exception, first: Forked JVM ran out of memory. See process stdout at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J2-20181114_113703_3986370795574671270387.sysout See process stderr at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J2-20181114_113703_3986118956146383826178.syserr

Total time: 584 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[Fast Archiver] No artifacts from Lucene-Solr-BadApples-NightlyTests-7.x #34 to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-NightlyTests-7.x - Build # 36 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-7.x/36/

9 tests failed.
FAILED:  org.apache.solr.cloud.BasicDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:40019/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:40019/collection1
        at __randomizedtesting.SeedInfo.seed([57FF9BDCA1BA41A:8D2BC66764E7C9E2]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.BaseDistributedSearchTestCase.add(BaseDistributedSearchTestCase.java:509)
        at org.apache.solr.BaseDistributedSearchTestCase.indexDoc(BaseDistributedSearchTestCase.java:498)
        at org.apache.solr.cloud.BasicDistributedZkTest.test(BasicDistributedZkTest.java:176)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 48 more


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
        at __randomizedtesting.SeedInfo.seed([57FF9BDCA1BA41A:A81F4DB6D7240C6F]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:102)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.api.collections.TestCollectionAPI.test

Error Message:
Error from server at <a href="http://127.0.0.1:39448:">http://127.0.0.1:39448: Could not load collection from ZK: collection1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at <a href="http://127.0.0.1:39448:">http://127.0.0.1:39448: Could not load collection from ZK: collection1
        at __randomizedtesting.SeedInfo.seed([57FF9BDCA1BA41A:8D2BC66764E7C9E2]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:392)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:341)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1008)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testDeleteNode

Error Message:
unexpected DELETENODE status: {responseHeader={status=0,QTime=50},status={state=notfound,msg=Did not find [search_rate_trigger3/5968c2c39a7ec4T20sf5nunah6c8ha1bzbocryf7/0] in any tasks queue}}

Stack Trace:
java.lang.AssertionError: unexpected DELETENODE status: {responseHeader={status=0,QTime=50},status={state=notfound,msg=Did not find [search_rate_trigger3/5968c2c39a7ec4T20sf5nunah6c8ha1bzbocryf7/0] in any tasks queue}}
        at __randomizedtesting.SeedInfo.seed([57FF9BDCA1BA41A:27ED373FFDD12B67]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.lambda$testDeleteNode$5(SearchRateTriggerIntegrationTest.java:693)
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testDeleteNode(SearchRateTriggerIntegrationTest.java:677)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testBelowSearchRate

Error Message:
[Op{action=DELETEREPLICA, hints={COLL_SHARD=[{   "first":"belowRate_collection",   "second":"shard1"}], REPLICA=[core_node10]}}, Op{action=DELETEREPLICA, hints={COLL_SHARD=[{   "first":"belowRate_collection",   "second":"shard1"}], REPLICA=[core_node8]}}, Op{action=DELETEREPLICA, hints={COLL_SHARD=[{   "first":"belowRate_collection",   "second":"shard1"}], REPLICA=[core_node6]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:43171_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:44199_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:44543_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:38466_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:45606_solr]}}] expected:<7> but was:<8>

Stack Trace:
java.lang.AssertionError: [Op{action=DELETEREPLICA, hints={COLL_SHARD=[{
  "first":"belowRate_collection",
  "second":"shard1"}], REPLICA=[core_node10]}}, Op{action=DELETEREPLICA, hints={COLL_SHARD=[{
  "first":"belowRate_collection",
  "second":"shard1"}], REPLICA=[core_node8]}}, Op{action=DELETEREPLICA, hints={COLL_SHARD=[{
  "first":"belowRate_collection",
  "second":"shard1"}], REPLICA=[core_node6]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:43171_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:44199_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:44543_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:38466_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:45606_solr]}}] expected:<7> but was:<8>
        at __randomizedtesting.SeedInfo.seed([57FF9BDCA1BA41A:6FD25FDD92F32109]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testBelowSearchRate(SearchRateTriggerIntegrationTest.java:400)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.search.TestRecovery.testExistOldBufferLog

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([57FF9BDCA1BA41A:5B2FE4E844D43493]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertFalse(Assert.java:68)
        at org.junit.Assert.assertFalse(Assert.java:79)
        at org.apache.solr.search.TestRecovery.testExistOldBufferLog(TestRecovery.java:1073)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testVersionsAreReturned

Error Message:
Error from server at https://127.0.0.1:41529/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason: <pre>    Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>  </body> </html>

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://127.0.0.1:41529/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at __randomizedtesting.SeedInfo.seed([C3F9ACD20CE5BA4D:3B3F55E79FFC2285]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:237)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testVersionsAreReturned(CloudSolrClientTest.java:725)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:41529/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testParallelUpdateQTime

Error Message:
Error from server at https://127.0.0.1:41529/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason: <pre>    Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>  </body> </html>

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://127.0.0.1:41529/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at __randomizedtesting.SeedInfo.seed([C3F9ACD20CE5BA4D:2D21D743429A6FDE]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testParallelUpdateQTime(CloudSolrClientTest.java:146)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:41529/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more


FAILED:  org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup

Error Message:
missing cleanup event: [CapturedEvent{timestamp=25166738965410798, stage=STARTED, actionName='null', event={   "id":"59690400f4498aT20sf5nunah6c8ha1bzbocryho",   "source":".scheduled_maintenance",   "eventTime":25166738844043658,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829337905,     "_enqueue_time_":25166738944432155}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166738981646956, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={   "id":"59690400f4498aT20sf5nunah6c8ha1bzbocryho",   "source":".scheduled_maintenance",   "eventTime":25166738844043658,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829337905,     "_enqueue_time_":25166738944432155}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166739041393505, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={   "id":"59690400f4498aT20sf5nunah6c8ha1bzbocryho",   "source":".scheduled_maintenance",   "eventTime":25166738844043658,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829337905,     "_enqueue_time_":25166738944432155}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.inactive_shard_plan={staleLocks={ScheduledMaintenanceTriggerTest_collection1/staleShard-splitting={stateTimestamp=1542656537785888372, currentTimeNs=1542829338103014242, deltaSec=172800, ttlSec=20}}}, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166739060044755, stage=SUCCEEDED, actionName='null', event={   "id":"59690400f4498aT20sf5nunah6c8ha1bzbocryho",   "source":".scheduled_maintenance",   "eventTime":25166738844043658,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829337905,     "_enqueue_time_":25166738944432155}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166744068988705, stage=STARTED, actionName='null', event={   "id":"596905384b96c5T20sf5nunah6c8ha1bzbocryhs",   "source":".scheduled_maintenance",   "eventTime":25166744067479237,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829343129,     "_enqueue_time_":25166744067671487}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166744069598603, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={   "id":"596905384b96c5T20sf5nunah6c8ha1bzbocryhs",   "source":".scheduled_maintenance",   "eventTime":25166744067479237,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829343129,     "_enqueue_time_":25166744067671487}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166744104944705, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={   "id":"596905384b96c5T20sf5nunah6c8ha1bzbocryhs",   "source":".scheduled_maintenance",   "eventTime":25166744067479237,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829343129,     "_enqueue_time_":25166744067671487}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166744106149025, stage=SUCCEEDED, actionName='null', event={   "id":"596905384b96c5T20sf5nunah6c8ha1bzbocryhs",   "source":".scheduled_maintenance",   "eventTime":25166744067479237,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829343129,     "_enqueue_time_":25166744067671487}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166749108435080, stage=STARTED, actionName='null', event={   "id":"59690664b6926cT20sf5nunah6c8ha1bzbocryhv",   "source":".scheduled_maintenance",   "eventTime":25166749107655276,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829348169,     "_enqueue_time_":25166749107857381}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166749109018548, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={   "id":"59690664b6926cT20sf5nunah6c8ha1bzbocryhv",   "source":".scheduled_maintenance",   "eventTime":25166749107655276,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829348169,     "_enqueue_time_":25166749107857381}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166749111623287, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={   "id":"59690664b6926cT20sf5nunah6c8ha1bzbocryhv",   "source":".scheduled_maintenance",   "eventTime":25166749107655276,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829348169,     "_enqueue_time_":25166749107857381}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166749112638559, stage=SUCCEEDED, actionName='null', event={   "id":"59690664b6926cT20sf5nunah6c8ha1bzbocryhv",   "source":".scheduled_maintenance",   "eventTime":25166749107655276,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829348169,     "_enqueue_time_":25166749107857381}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166754126528340, stage=STARTED, actionName='null', event={   "id":"5969078fd04bf1T20sf5nunah6c8ha1bzbocryhw",   "source":".scheduled_maintenance",   "eventTime":25166754125728753,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829353187,     "_enqueue_time_":25166754125978299}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166754127123921, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={   "id":"5969078fd04bf1T20sf5nunah6c8ha1bzbocryhw",   "source":".scheduled_maintenance",   "eventTime":25166754125728753,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829353187,     "_enqueue_time_":25166754125978299}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166754128738301, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={   "id":"5969078fd04bf1T20sf5nunah6c8ha1bzbocryhw",   "source":".scheduled_maintenance",   "eventTime":25166754125728753,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829353187,     "_enqueue_time_":25166754125978299}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166754179113656, stage=SUCCEEDED, actionName='null', event={   "id":"5969078fd04bf1T20sf5nunah6c8ha1bzbocryhw",   "source":".scheduled_maintenance",   "eventTime":25166754125728753,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829353187,     "_enqueue_time_":25166754125978299}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166759184486585, stage=STARTED, actionName='null', event={   "id":"596908bd49e6cdT20sf5nunah6c8ha1bzbocryhx",   "source":".scheduled_maintenance",   "eventTime":25166759183640269,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829358245,     "_enqueue_time_":25166759183920143}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166759185058841, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={   "id":"596908bd49e6cdT20sf5nunah6c8ha1bzbocryhx",   "source":".scheduled_maintenance",   "eventTime":25166759183640269,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829358245,     "_enqueue_time_":25166759183920143}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166759189433867, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={   "id":"596908bd49e6cdT20sf5nunah6c8ha1bzbocryhx",   "source":".scheduled_maintenance",   "eventTime":25166759183640269,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829358245,     "_enqueue_time_":25166759183920143}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166759190525077, stage=SUCCEEDED, actionName='null', event={   "id":"596908bd49e6cdT20sf5nunah6c8ha1bzbocryhx",   "source":".scheduled_maintenance",   "eventTime":25166759183640269,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829358245,     "_enqueue_time_":25166759183920143}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166764201624760, stage=STARTED, actionName='null', event={   "id":"596909e856c668T20sf5nunah6c8ha1bzbocryhy",   "source":".scheduled_maintenance",   "eventTime":25166764200871528,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829363263,     "_enqueue_time_":25166764201069957}}, context={}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166764202186191, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={   "id":"596909e856c668T20sf5nunah6c8ha1bzbocryhy",   "source":".scheduled_maintenance",   "eventTime":25166764200871528,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829363263,     "_enqueue_time_":25166764201069957}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166764203471345, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={   "id":"596909e856c668T20sf5nunah6c8ha1bzbocryhy",   "source":".scheduled_maintenance",   "eventTime":25166764200871528,   "eventType":"SCHEDULED",   "properties":{     "actualEventTime":1542829363263,     "_enqueue_time_":25166764201069957}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.inactive_shard_plan={staleLocks={ScheduledMaintenanceTriggerTest_collection1/shard1-splitting={stateTimestamp=1542829338076781855, currentTimeNs=1542829363265180389, deltaSec=25, ttlSec=20}}}, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={   "trigger":".scheduled_maintenance",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}]

Stack Trace:
java.lang.AssertionError: missing cleanup event: [CapturedEvent{timestamp=25166738965410798, stage=STARTED, actionName='null', event={
  "id":"59690400f4498aT20sf5nunah6c8ha1bzbocryho",
  "source":".scheduled_maintenance",
  "eventTime":25166738844043658,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829337905,
    "_enqueue_time_":25166738944432155}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166738981646956, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={
  "id":"59690400f4498aT20sf5nunah6c8ha1bzbocryho",
  "source":".scheduled_maintenance",
  "eventTime":25166738844043658,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829337905,
    "_enqueue_time_":25166738944432155}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166739041393505, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={
  "id":"59690400f4498aT20sf5nunah6c8ha1bzbocryho",
  "source":".scheduled_maintenance",
  "eventTime":25166738844043658,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829337905,
    "_enqueue_time_":25166738944432155}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.inactive_shard_plan={staleLocks={ScheduledMaintenanceTriggerTest_collection1/staleShard-splitting={stateTimestamp=1542656537785888372, currentTimeNs=1542829338103014242, deltaSec=172800, ttlSec=20}}}, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166739060044755, stage=SUCCEEDED, actionName='null', event={
  "id":"59690400f4498aT20sf5nunah6c8ha1bzbocryho",
  "source":".scheduled_maintenance",
  "eventTime":25166738844043658,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829337905,
    "_enqueue_time_":25166738944432155}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166744068988705, stage=STARTED, actionName='null', event={
  "id":"596905384b96c5T20sf5nunah6c8ha1bzbocryhs",
  "source":".scheduled_maintenance",
  "eventTime":25166744067479237,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829343129,
    "_enqueue_time_":25166744067671487}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166744069598603, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={
  "id":"596905384b96c5T20sf5nunah6c8ha1bzbocryhs",
  "source":".scheduled_maintenance",
  "eventTime":25166744067479237,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829343129,
    "_enqueue_time_":25166744067671487}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166744104944705, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={
  "id":"596905384b96c5T20sf5nunah6c8ha1bzbocryhs",
  "source":".scheduled_maintenance",
  "eventTime":25166744067479237,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829343129,
    "_enqueue_time_":25166744067671487}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166744106149025, stage=SUCCEEDED, actionName='null', event={
  "id":"596905384b96c5T20sf5nunah6c8ha1bzbocryhs",
  "source":".scheduled_maintenance",
  "eventTime":25166744067479237,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829343129,
    "_enqueue_time_":25166744067671487}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166749108435080, stage=STARTED, actionName='null', event={
  "id":"59690664b6926cT20sf5nunah6c8ha1bzbocryhv",
  "source":".scheduled_maintenance",
  "eventTime":25166749107655276,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829348169,
    "_enqueue_time_":25166749107857381}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166749109018548, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={
  "id":"59690664b6926cT20sf5nunah6c8ha1bzbocryhv",
  "source":".scheduled_maintenance",
  "eventTime":25166749107655276,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829348169,
    "_enqueue_time_":25166749107857381}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166749111623287, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={
  "id":"59690664b6926cT20sf5nunah6c8ha1bzbocryhv",
  "source":".scheduled_maintenance",
  "eventTime":25166749107655276,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829348169,
    "_enqueue_time_":25166749107857381}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166749112638559, stage=SUCCEEDED, actionName='null', event={
  "id":"59690664b6926cT20sf5nunah6c8ha1bzbocryhv",
  "source":".scheduled_maintenance",
  "eventTime":25166749107655276,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829348169,
    "_enqueue_time_":25166749107857381}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166754126528340, stage=STARTED, actionName='null', event={
  "id":"5969078fd04bf1T20sf5nunah6c8ha1bzbocryhw",
  "source":".scheduled_maintenance",
  "eventTime":25166754125728753,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829353187,
    "_enqueue_time_":25166754125978299}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166754127123921, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={
  "id":"5969078fd04bf1T20sf5nunah6c8ha1bzbocryhw",
  "source":".scheduled_maintenance",
  "eventTime":25166754125728753,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829353187,
    "_enqueue_time_":25166754125978299}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166754128738301, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={
  "id":"5969078fd04bf1T20sf5nunah6c8ha1bzbocryhw",
  "source":".scheduled_maintenance",
  "eventTime":25166754125728753,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829353187,
    "_enqueue_time_":25166754125978299}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166754179113656, stage=SUCCEEDED, actionName='null', event={
  "id":"5969078fd04bf1T20sf5nunah6c8ha1bzbocryhw",
  "source":".scheduled_maintenance",
  "eventTime":25166754125728753,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829353187,
    "_enqueue_time_":25166754125978299}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166759184486585, stage=STARTED, actionName='null', event={
  "id":"596908bd49e6cdT20sf5nunah6c8ha1bzbocryhx",
  "source":".scheduled_maintenance",
  "eventTime":25166759183640269,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829358245,
    "_enqueue_time_":25166759183920143}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166759185058841, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={
  "id":"596908bd49e6cdT20sf5nunah6c8ha1bzbocryhx",
  "source":".scheduled_maintenance",
  "eventTime":25166759183640269,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829358245,
    "_enqueue_time_":25166759183920143}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166759189433867, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={
  "id":"596908bd49e6cdT20sf5nunah6c8ha1bzbocryhx",
  "source":".scheduled_maintenance",
  "eventTime":25166759183640269,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829358245,
    "_enqueue_time_":25166759183920143}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166759190525077, stage=SUCCEEDED, actionName='null', event={
  "id":"596908bd49e6cdT20sf5nunah6c8ha1bzbocryhx",
  "source":".scheduled_maintenance",
  "eventTime":25166759183640269,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829358245,
    "_enqueue_time_":25166759183920143}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166764201624760, stage=STARTED, actionName='null', event={
  "id":"596909e856c668T20sf5nunah6c8ha1bzbocryhy",
  "source":".scheduled_maintenance",
  "eventTime":25166764200871528,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829363263,
    "_enqueue_time_":25166764201069957}}, context={}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166764202186191, stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={
  "id":"596909e856c668T20sf5nunah6c8ha1bzbocryhy",
  "source":".scheduled_maintenance",
  "eventTime":25166764200871528,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829363263,
    "_enqueue_time_":25166764201069957}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}, CapturedEvent{timestamp=25166764203471345, stage=AFTER_ACTION, actionName='inactive_shard_plan', event={
  "id":"596909e856c668T20sf5nunah6c8ha1bzbocryhy",
  "source":".scheduled_maintenance",
  "eventTime":25166764200871528,
  "eventType":"SCHEDULED",
  "properties":{
    "actualEventTime":1542829363263,
    "_enqueue_time_":25166764201069957}}, context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], source=.scheduled_maintenance, properties.inactive_shard_plan={staleLocks={ScheduledMaintenanceTriggerTest_collection1/shard1-splitting={stateTimestamp=1542829338076781855, currentTimeNs=1542829363265180389, deltaSec=25, ttlSec=20}}}, properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={
  "trigger":".scheduled_maintenance",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "beforeAction":"inactive_shard_plan",
  "afterAction":"inactive_shard_plan",
  "class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"}, message='null'}]
        at __randomizedtesting.SeedInfo.seed([57FF9BDCA1BA41A:185339CFAB588311]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup(ScheduledMaintenanceTriggerTest.java:253)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13692 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SearchRateTriggerIntegrationTest_57FF9BDCA1BA41A-001/init-core-data-001
   [junit4]   2> 1000666 INFO  (SUITE-SearchRateTriggerIntegrationTest-seed#[57FF9BDCA1BA41A]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1000666 INFO  (SUITE-SearchRateTriggerIntegrationTest-seed#[57FF9BDCA1BA41A]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1000681 INFO  (SUITE-SearchRateTriggerIntegrationTest-seed#[57FF9BDCA1BA41A]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1000682 INFO  (SUITE-SearchRateTriggerIntegrationTest-seed#[57FF9BDCA1BA41A]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 5 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SearchRateTriggerIntegrationTest_57FF9BDCA1BA41A-001/tempDir-001
   [junit4]   2> 1000682 INFO  (SUITE-SearchRateTriggerIntegrationTest-seed#[57FF9BDCA1BA41A]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1000698 INFO  (Thread-35235) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1000698 INFO  (Thread-35235) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1000700 ERROR (Thread-35235) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1000807 INFO  (SUITE-SearchRateTriggerIntegrationTest-seed#[57FF9BDCA1BA41A]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:44533
   [junit4]   2> 1000844 INFO  (zkConnectionManagerCallback-864-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1000882 INFO  (jetty-launcher-861-thread-2) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
   [junit4]   2> 1000882 INFO  (jetty-launcher-861-thread-1) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
   [junit4]   2> 1000918 INFO  (jetty-launcher-861-thread-3) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
   [junit4]   2> 1000918 INFO  (jetty-launcher-861-thread-4) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
   [junit4]   2> 1000974 INFO  (jetty-launcher-861-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1000974 INFO  (jetty-launcher-861-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1000974 INFO  (jetty-lau

[...truncated too long message...]

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/null703860387
     [copy] Copying 241 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/null703860387
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/null703860387

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 37ms :: artifacts dl 4ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 634 minutes 48 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-NightlyTests-7.x - Build # 37 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-7.x/37/

2 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testEventQueue

Error Message:
action wasn't interrupted

Stack Trace:
java.lang.AssertionError: action wasn't interrupted
        at __randomizedtesting.SeedInfo.seed([BB5D3569FE6D602A:72E877C7F70AA6DF]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testEventQueue(TestSimTriggerIntegration.java:686)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting

Error Message:
Error from server at http://127.0.0.1:43566/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason: <pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>  </body> </html>

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://127.0.0.1:43566/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at __randomizedtesting.SeedInfo.seed([76ECABC04EB9D225:B45B97A84DF9225D]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:237)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting(CloudSolrClientTest.java:269)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:43566/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more




Build Log:
[...truncated 14844 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestSimTriggerIntegration_BB5D3569FE6D602A-001/init-core-data-001
   [junit4]   2> 4732085 INFO  (SUITE-TestSimTriggerIntegration-seed#[BB5D3569FE6D602A]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 4732087 INFO  (SUITE-TestSimTriggerIntegration-seed#[BB5D3569FE6D602A]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 4732088 INFO  (SUITE-TestSimTriggerIntegration-seed#[BB5D3569FE6D602A]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 4732098 DEBUG (SUITE-TestSimTriggerIntegration-seed#[BB5D3569FE6D602A]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10015_solr
   [junit4]   2> 4732099 DEBUG (SUITE-TestSimTriggerIntegration-seed#[BB5D3569FE6D602A]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=0
   [junit4]   2> 4732099 DEBUG (SUITE-TestSimTriggerIntegration-seed#[BB5D3569FE6D602A]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
   [junit4]   2> 4732099 INFO  (SUITE-TestSimTriggerIntegration-seed#[BB5D3569FE6D602A]-worker) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4732101 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 4732101 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
   [junit4]   2> 4732112 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
   [junit4]   2> 4732112 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
   [junit4]   2> 4732118 INFO  (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testEventQueue
   [junit4]   2> 4732118 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 4732118 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10015_solr]
   [junit4]   2> 4732119 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 4732119 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
   [junit4]   2> 4732119 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 4732119 INFO  (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 4732123 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 4732123 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 4732123 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 4732124 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 4732124 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 4732124 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 4732126 WARN  (ScheduledTrigger-11880-thread-1) [    ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger ran but was already closed
   [junit4]   2> 4732124 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_191]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:177) [java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4732126 ERROR (ScheduledTrigger-11880-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Unexpected exception from trigger: .scheduled_maintenance
   [junit4]   2> java.lang.RuntimeException: Trigger has been closed
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTrigger.run(ScheduledTrigger.java:155) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.run(ScheduledTriggers.java:592) [java/:?]
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4732126 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 4732127 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 4732127 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 4732127 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 4732128 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
   [junit4]   2> 4732128 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 4732128 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10015_solr]
   [junit4]   2> 4732128 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 4732128 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 4732129 DEBUG (ScheduledTrigger-11885-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732151 DEBUG (ScheduledTrigger-11885-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732172 DEBUG (ScheduledTrigger-11885-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732201 DEBUG (ScheduledTrigger-11885-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732222 DEBUG (ScheduledTrigger-11885-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732242 DEBUG (ScheduledTrigger-11885-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732262 DEBUG (ScheduledTrigger-11885-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732283 DEBUG (ScheduledTrigger-11885-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732303 DEBUG (ScheduledTrigger-11885-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732323 DEBUG (ScheduledTrigger-11885-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732328 DEBUG (simCloudManagerPool-11884-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=1
   [junit4]   2> 4732328 DEBUG (simCloudManagerPool-11884-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 1
   [junit4]   2> 4732329 DEBUG (simCloudManagerPool-11884-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 4732329 DEBUG (simCloudManagerPool-11884-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 4732332 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 4732339 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 4732339 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 4732361 DEBUG (ScheduledTrigger-11885-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732381 DEBUG (ScheduledTrigger-11885-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732402 DEBUG (ScheduledTrigger-11885-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732422 DEBUG (ScheduledTrigger-11885-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732442 DEBUG (ScheduledTrigger-11885-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732463 DEBUG (ScheduledTrigger-11885-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732483 DEBUG (ScheduledTrigger-11885-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732503 DEBUG (ScheduledTrigger-11885-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732524 DEBUG (ScheduledTrigger-11885-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732533 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection .system, currentVersion=2
   [junit4]   2> 4732533 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=2
   [junit4]   2> 4732533 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 2
   [junit4]   2> 4732543 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=3
   [junit4]   2> 4732544 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 3
   [junit4]   2> 4732544 DEBUG (ScheduledTrigger-11885-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732556 DEBUG (simCloudManagerPool-11884-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=4
   [junit4]   2> 4732556 DEBUG (simCloudManagerPool-11884-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 4
   [junit4]   2> 4732557 DEBUG (simCloudManagerPool-11884-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1 (currentVersion=5): {"core_node1":{
   [junit4]   2>     "core":".system_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":".system",
   [junit4]   2>     "node_name":"127.0.0.1:10015_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 4732564 DEBUG (ScheduledTrigger-11885-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732567 DEBUG (simCloudManagerPool-11884-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=5
   [junit4]   2> 4732567 DEBUG (simCloudManagerPool-11884-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 5
   [junit4]   2> 4732567 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection .system, currentVersion=6
   [junit4]   2> 4732570 INFO  (simCloudManagerPool-11884-thread-5) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestEventQueueAction instantiated
   [junit4]   2> 4732570 DEBUG (simCloudManagerPool-11884-thread-5) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 4732571 DEBUG (simCloudManagerPool-11884-thread-5) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 4732571 DEBUG (simCloudManagerPool-11884-thread-5) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 4732571 INFO  (simCloudManagerPool-11884-thread-5) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestEventQueueAction instantiated
   [junit4]   2> 4732571 DEBUG (simCloudManagerPool-11884-thread-5) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 4732571 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 4732572 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 - Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10015_solr]
   [junit4]   2> 4732572 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestEventQueueAction}], enabled=true}
   [junit4]   2> 4732572 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 4732572 DEBUG (ScheduledTrigger-11885-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 4732573 DEBUG (ScheduledTrigger-11885-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 4732585 DEBUG (ScheduledTrigger-11885-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732593 DEBUG (ScheduledTrigger-11885-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 4732593 DEBUG (ScheduledTrigger-11885-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 4732605 DEBUG (ScheduledTrigger-11885-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732614 DEBUG (ScheduledTrigger-11885-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 4732614 DEBUG (ScheduledTrigger-11885-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 4732614 DEBUG (ScheduledTrigger-11885-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10017_solr at time 4791581053827263
   [junit4]   2> 4732626 DEBUG (ScheduledTrigger-11885-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 4732645 DEBUG (ScheduledTrigger-11885-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 4732645 DEBUG (ScheduledTrigger-11885-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 4732645 DEBUG (ScheduledTrigger-11885-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing registered processor for nodes: [127.0.0.1:10017_solr] added at times [4791581053827263], now=4791582598716413
   [junit4]   2> 4732645 DEBUG (ScheduledTrigger-11885-thread-2) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"1105eae6a038bfT84uj8qni1ikeu31q71tqcgaog",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":4791581053827263,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[4791581053827263],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "nodeNames":["127.0.0.1:10017_solr"]}}
   [junit4]   2> 4732648 DEBUG (ScheduledTrigger-11885-thread-2) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_trigger1]
   [junit4]   2> 4732682 DEBUG (ScheduledTrigger-11885-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 - removing marker path: /autoscaling/nodeAdded/127.0.0.1:10017_solr
   [junit4]   2> 4732683 DEBUG (AutoscalingActionExecutor-11886-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"1105eae6a038bfT84uj8qni1ikeu31q71tqcgaog",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":4791581053827263,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[4791581053827263],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":4791582742741963,
   [junit4]   2>     "nodeNames":["127.0.0.1:10017_solr"]}}
   [junit4]   2> 4732683 INFO  (AutoscalingActionExecutor-11886-thread-1) [    ] o.a.s.c.a.s.TestSimTriggerIntegration -- event: {
   [junit4]   2>   "id":"1105eae6a038bfT84uj8qni1ikeu31q71tqcgaog",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":4791581053827263,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[4791581053827263],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":4791582742741963,
   [junit4]   2>     "nodeNames":["127.0.0.1:10017_solr"]}}
   [junit4]   2> 4732683 INFO  (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 4732683 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 4732697 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 4732697 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 4732697 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 4732706 WARN  (AutoscalingActionExecutor-11886-thread-1) [    ] o.a.s.c.a.SystemLogListener Exception sending event to collection .system
   [junit4]   2> java.io.IOException: java.lang.InterruptedException
   [junit4]   2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:654) ~[test/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.SystemLogListener.onEvent(SystemLogListener.java:118) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:793) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:760) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:326) ~[java/:?]
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_191]
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> Caused by: java.lang.InterruptedException
   [junit4]   2> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) ~[?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:1.8.0_191]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:652) ~[test/:?]
   [junit4]   2> ... 10 more
   [junit4]   2> 4732707 WARN  (AutoscalingActionExecutor-11886-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Unhandled exception executing actions
   [junit4]   2> org.apache.lucene.store.AlreadyClosedException: ScheduledTrigger node_added_trigger1 has been closed.
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.dequeue(ScheduledTriggers.java:549) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:319) ~[java/:?]
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_191]
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4732708 DEBUG (AutoscalingActionExecutor-11886-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 1259 ms for event id=1105eae6a038bfT84uj8qni1ikeu31q71tqcgaog
   [junit4]   2> 4732708 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 4732708 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 4732708 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 4732708 INFO  (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimCloudManager   = killing node 127.0.0.1:10015_solr
   [junit4]   2> 4732708 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10016_solr
   [junit4]   2> 4732708 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=6
   [junit4]   2> 4732708 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 6
   [junit4]   2> 4732728 DEBUG (simCloudManagerPool-11884-thread-8) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1 (currentVersion=7): {"core_node2":{
   [junit4]   2>     "core":".system_shard1_replica_n2",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":".system",
   [junit4]   2>     "node_name":"127.0.0.1:10016_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":1,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10740,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":1.000240445137024E-5,
   [junit4]   2>     "SEARCHER.searcher.numDocs":1}}
   [junit4]   2> 4732729 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 4732730 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestEventQueueAction instantiated
   [junit4]   2> 4732730 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 4732730 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion -1
   [junit4]   2> 4732730 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 4732730 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10017_solr]
   [junit4]   2> 4732730 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:10015_solr
   [junit4]   2> 4732741 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 - Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10017_solr]
   [junit4]   2> 4732741 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestEventQueueAction}], enabled=true}
   [junit4]   2> 4732741 DEBUG (ScheduledTrigger-11888-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732741 DEBUG (ScheduledTrigger-11888-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10015_solr
   [junit4]   2> 4732749 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 4732749 DEBUG (ScheduledTrigger-11888-thread-2) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"1105eae6a038bfT84uj8qni1ikeu31q71tqcgaog",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":4791581053827263,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[4791581053827263],
   [junit4]   2>     "_dequeue_time_":4791587809871063,
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":4791582742741963,
   [junit4]   2>     "nodeNames":["127.0.0.1:10017_solr"],
   [junit4]   2>     "replaying":true}}
   [junit4]   2> 4732775 DEBUG (ScheduledTrigger-11888-thread-2) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_trigger1]
   [junit4]   2> 4732797 DEBUG (simCloudManagerPool-11887-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=7
   [junit4]   2> 4732797 DEBUG (simCloudManagerPool-11887-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 7
   [junit4]   2> 4732798 DEBUG (ScheduledTrigger-11888-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 4732798 DEBUG (ScheduledTrigger-11888-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 4732798 DEBUG (AutoscalingActionExecutor-11889-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"1105eae6a038bfT84uj8qni1ikeu31q71tqcgaog",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":4791581053827263,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[4791581053827263],
   [junit4]   2>     "_dequeue_time_":4791587809871063,
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":4791582742741963,
   [junit4]   2>     "nodeNames":["127.0.0.1:10017_solr"],
   [junit4]   2>     "replaying":true}}
   [junit4]   2> 4732798 INFO  (AutoscalingActionExecutor-11889-thread-1) [    ] o.a.s.c.a.s.TestSimTriggerIntegration -- event: {
   [junit4]   2>   "id":"1105eae6a038bfT84uj8qni1ikeu31q71tqcgaog",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":4791581053827263,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[4791581053827263],
   [junit4]   2>     "_dequeue_time_":4791587809871063,
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":4791582742741963,
   [junit4]   2>     "nodeNames":["127.0.0.1:10017_solr"],
   [junit4]   2>     "replaying":true}}
   [junit4]   2> 4732822 DEBUG (AutoscalingActionExecutor-11889-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 4732822 DEBUG (AutoscalingActionExecutor-11889-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger1 after 100ms
   [junit4]   2> 4732822 DEBUG (AutoscalingActionExecutor-11889-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 1205 ms for event id=1105eae6a038bfT84uj8qni1ikeu31q71tqcgaog
   [junit4]   2> 4732890 INFO  (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testEventQueue
   [junit4]   2> 4732890 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=8
   [junit4]   2> 4732890 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 8
   [junit4]   2> 4732891 INFO  (TEST-TestSimTriggerIntegration.testEventQueue-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes: 2
   [junit4]   2> ## Empty nodes: 1
   [junit4]   2> ## Dead nodes: 1
   [junit4]   2> ## 127.0.0.1:10015_solr
   [junit4]   2> ## Collections:
   [junit4]   2> ##  * .system
   [junit4]   2> ##    shardsTotal 1
   [junit4]   2> ##    shardsState {active=1}
   [junit4]   2> ##      shardsWithoutLeader 0
   [junit4]   2> ##    totalReplicas 2
   [junit4]   2> ##      activeReplicas 2
   [junit4]   2> ##      inactiveReplicas 0
   [junit4]   2> ##    totalActiveDocs 3
   [junit4]   2> ##      bufferedDocs 0
   [junit4]   2> ##      maxActiveSliceDocs 3
   [junit4]   2> ##      minActiveSliceDocs 3
   [junit4]   2> ##      avgActiveSliceDocs 3
   [junit4]   2> ##    totalInactiveDocs 0
   [junit4]   2> ##      maxInactiveSliceDocs 0
   [junit4]   2> ##      minInactiveSliceDocs 0
   [junit4]   2> ##      avgInactiveSliceDocs NaN
   [junit4]   2> ##    totalActiveBytes 11,740
   [junit4]   2> ##      maxActiveSliceBytes 11,740
   [junit4]   2> ##      minActiveSliceBytes 11,740
   [junit4]   2> ##      avgActiveSliceBytes 11,740
   [junit4]   2> ##    totalInactiveBytes 0
   [junit4]   2> ##      maxInactiveSliceBytes 0
   [junit4]   2> ##      minInactiveSliceBytes 0
   [junit4]   2> ##      avgInactiveSliceBytes NaN
   [junit4]   2> ##    totalActiveDeletedDocs 0
   [junit4]   2> ## Max replicas per node: 1
   [junit4]   2> ## Min replicas per node: 0
   [junit4]   2> ## Total replicas: 1
   [junit4]   2> ## * .system 1
   [junit4]   2> ## - active           1
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ## - autoscaling        2
   [junit4]   2> ## - update             4
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: node_added_trigger1
   [junit4]   2> ## - STARTED         2
   [junit4]   2> ## - SUCCEEDED       1
   [junit4]   2>
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSimTriggerIntegration -Dtests.method=testEventQueue -Dtests.seed=BB5D3569FE6D602A -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=fr-CA -Dtests.timezone=Antarctica/Palmer -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 0.79s J1 | TestSimTriggerIntegration.testEventQueue <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: action wasn't interrupted
   [junit4]    > at __randomizedtesting.SeedInfo.seed([BB5D3569FE6D602A:72E877C7F70AA6DF]:0)
   [junit4]    > at org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testEventQueue(TestSimTriggerIntegration.java:686)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 4732926 DEBUG (ScheduledTrigger-11888-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732928 DEBUG (ScheduledTrigger-11888-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 4732928 DEBUG (ScheduledTrigger-11888-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 4732930 INFO  (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeMarkersRegistration
   [junit4]   2> 4732931 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 4732931 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10015_solr
   [junit4]   2> 4732931 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 4732931 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 4732931 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 4732931 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 4732933 INFO  (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 4732933 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 4732936 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 4732936 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 4732936 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 4732936 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 4732936 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 4732936 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 4732937 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 4732937 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 4732938 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion -1
   [junit4]   2> 4732938 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 4732938 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10017_solr]
   [junit4]   2> 4732939 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 4732939 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 4732939 DEBUG (ScheduledTrigger-11891-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732963 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4732984 DEBUG (ScheduledTrigger-11891-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733004 DEBUG (ScheduledTrigger-11891-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733024 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733045 DEBUG (ScheduledTrigger-11891-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733065 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733086 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733106 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733126 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733138 DEBUG (simCloudManagerPool-11890-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=9
   [junit4]   2> 4733138 DEBUG (simCloudManagerPool-11890-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 9
   [junit4]   2> 4733138 DEBUG (simCloudManagerPool-11890-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 4733139 DEBUG (simCloudManagerPool-11890-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 4733141 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 4733142 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 4733142 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 4733146 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733167 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733187 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733207 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733228 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733248 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733269 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733289 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733310 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733330 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733342 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection .system, currentVersion=10
   [junit4]   2> 4733342 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=10
   [junit4]   2> 4733342 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 10
   [junit4]   2> 4733351 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733353 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=11
   [junit4]   2> 4733353 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 11
   [junit4]   2> 4733365 DEBUG (simCloudManagerPool-11890-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=12
   [junit4]   2> 4733365 DEBUG (simCloudManagerPool-11890-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 12
   [junit4]   2> 4733366 DEBUG (simCloudManagerPool-11890-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1 (currentVersion=13): {"core_node3":{
   [junit4]   2>     "core":".system_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":".system",
   [junit4]   2>     "node_name":"127.0.0.1:10017_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 4733371 DEBUG (ScheduledTrigger-11891-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733376 DEBUG (simCloudManagerPool-11890-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=13
   [junit4]   2> 4733377 DEBUG (simCloudManagerPool-11890-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 13
   [junit4]   2> 4733377 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection .system, currentVersion=14
   [junit4]   2> 4733378 INFO  (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.TestSimTriggerIntegration ====== KILL OVERSEER 1
   [junit4]   2> 4733378 INFO  (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 4733381 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 4733379 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_191]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189) [java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> 4733384 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 4733384 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 4733384 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 4733384 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 4733384 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 4733384 INFO  (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimCloudManager   = killing node 127.0.0.1:10017_solr
   [junit4]   2> 4733384 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=14
   [junit4]   2> 4733385 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 14
   [junit4]   2> 4733388 INFO  (simCloudManagerPool-11890-thread-5) [    ] o.a.s.c.ActionThrottle Throttling leader attempts - waiting for 3899ms
   [junit4]   2> 4733388 DEBUG (simCloudManagerPool-11890-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1 (currentVersion=15): {"core_node4":{
   [junit4]   2>     "core":".system_shard1_replica_n2",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":".system",
   [junit4]   2>     "node_name":"127.0.0.1:10016_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 4733389 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 4733390 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion -1
   [junit4]   2> 4733390 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 4733391 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10018_solr, 127.0.0.1:10016_solr]
   [junit4]   2> 4733391 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:10017_solr
   [junit4]   2> 4733391 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 4733391 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 4733391 DEBUG (ScheduledTrigger-11894-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733391 DEBUG (ScheduledTrigger-11894-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10017_solr
   [junit4]   2> 4733412 DEBUG (ScheduledTrigger-11894-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733436 DEBUG (ScheduledTrigger-11894-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733460 DEBUG (ScheduledTrigger-11894-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733480 DEBUG (ScheduledTrigger-11894-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733489 INFO  (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.TestSimTriggerIntegration ====== ADD TRIGGERS
   [junit4]   2> 4733494 INFO  (simCloudManagerPool-11893-thread-1) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 4733494 DEBUG (simCloudManagerPool-11893-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=15
   [junit4]   2> 4733495 DEBUG (simCloudManagerPool-11893-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 15
   [junit4]   2> 4733495 DEBUG (simCloudManagerPool-11893-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 4733495 DEBUG (simCloudManagerPool-11893-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
   [junit4]   2> 4733497 INFO  (simCloudManagerPool-11893-thread-1) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 4733498 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 4733498 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - Initial livenodes: [127.0.0.1:10018_solr, 127.0.0.1:10016_solr]
   [junit4]   2> 4733498 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestEventMarkerAction}], enabled=true}
   [junit4]   2> 4733498 DEBUG (ScheduledTrigger-11894-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 4733498 DEBUG (ScheduledTrigger-11894-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 4733499 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 4733499 INFO  (simCloudManagerPool-11893-thread-2) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 4733499 DEBUG (simCloudManagerPool-11893-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 4733500 DEBUG (simCloudManagerPool-11893-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
   [junit4]   2> 4733500 INFO  (simCloudManagerPool-11893-thread-2) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 4733500 INFO  (simCloudManagerPool-11893-thread-2) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 4733500 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 4733501 DEBUG (ScheduledTrigger-11894-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733501 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_trigger - Initial livenodes: [127.0.0.1:10018_solr, 127.0.0.1:10016_solr]
   [junit4]   2> 4733501 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 4733501 DEBUG (ScheduledTrigger-11894-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
   [junit4]   2> 4733501 INFO  (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.TestSimTriggerIntegration ====== ADD NODE 1
   [junit4]   2> 4733519 DEBUG (ScheduledTrigger-11894-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 4733519 DEBUG (ScheduledTrigger-11894-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 4733519 DEBUG (ScheduledTrigger-11894-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10019_solr at time 4791626305389663
   [junit4]   2> 4733521 DEBUG (ScheduledTrigger-11894-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 4733521 DEBUG (ScheduledTrigger-11894-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 3
   [junit4]   2> 4733544 DEBUG (ScheduledTrigger-11894-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 4733542 DEBUG (ScheduledTrigger-11894-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 4733544 DEBUG (ScheduledTrigger-11894-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 4733544 DEBUG (ScheduledTrigger-11894-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10019_solr] added at times [4791626305389663], now=4791627539321863
   [junit4]   2> 4733544 DEBUG (ScheduledTrigger-11894-thread-1) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"1105f56fd4445fT84uj8qni1ikeu31q71tqcgaol",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":4791626305389663,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[4791626305389663],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "nodeNames":["127.0.0.1:10019_solr"]}}
   [junit4]   2> 4733546 DEBUG (ScheduledTrigger-11894-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 3
   [junit4]   2> 4733546 DEBUG (ScheduledTrigger-11894-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_lost_trigger, node_added_trigger]
   [junit4]   2> 4733547 DEBUG (ScheduledTrigger-11894-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - removing marker path: /autoscaling/nodeAdded/127.0.0.1:10019_solr
   [junit4]   2> 4733548 DEBUG (AutoscalingActionExecutor-11895-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"1105f56fd4445fT84uj8qni1ikeu31q71tqcgaol",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":4791626305389663,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[4791626305389663],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":4791627640495013,
   [junit4]   2>     "nodeNames":["127.0.0.1:10019_solr"]}}
   [junit4]   2> 4733549 DEBUG (AutoscalingActionExecutor-11895-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 4733549 DEBUG (AutoscalingActionExecutor-11895-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger after 100ms
   [junit4]   2> 4733549 DEBUG (AutoscalingActionExecutor-11895-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
   [junit4]   2> 4733549 DEBUG (AutoscalingActionExecutor-11895-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 74 ms for event id=1105f56fd4445fT84uj8qni1ikeu31q71tqcgaol
   [junit4]   2> 4733602 INFO  (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.TestSimTriggerIntegration ====== KILL OVERSEER 2
   [junit4]   2> 4733602 INFO  (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 4733602 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 4733602 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 4733604 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 4733604 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 4733605 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 4733605 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 4733605 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 4733605 INFO  (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimCloudManager   = killing node 127.0.0.1:10018_solr
   [junit4]   2> 4733606 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
   [junit4]   2> 4733606 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 4733606 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 4733606 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion -1
   [junit4]   2> 4733606 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 4733607 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_trigger - Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10019_solr]
   [junit4]   2> 4733607 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:10018_solr
   [junit4]   2> 4733607 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10019_solr]
   [junit4]   2> 4733607 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestEventMarkerAction}], enabled=true}
   [junit4]   2> 4733607 DEBUG (ScheduledTrigger-11897-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
   [junit4]   2> 4733607 DEBUG (ScheduledTrigger-11897-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10018_solr
   [junit4]   2> 4733607 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10019_solr]
   [junit4]   2> 4733608 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 4733608 DEBUG (ScheduledTrigger-11897-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733608 DEBUG (ScheduledTrigger-11897-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10018_solr
   [junit4]   2> 4733608 DEBUG (ScheduledTrigger-11897-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 4733608 DEBUG (ScheduledTrigger-11897-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 4733630 DEBUG (ScheduledTrigger-11897-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
   [junit4]   2> 4733632 DEBUG (ScheduledTrigger-11897-thread-3) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10018_solr]
   [junit4]   2> 4733632 DEBUG (ScheduledTrigger-11897-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4733634 DEBUG (ScheduledTrigger-11897-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 4733634 DEBUG (ScheduledTrigger-11897-thread-3) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"1105f676dec693T84uj8qni1ikeu31q71tqcgaoo",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":4791630718486163,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[4791630718486163],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "nodeNames":["127.0.0.1:10018_solr"]}}
   [junit4]   2> 4733634 DEBUG (ScheduledTrigger-11897-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 4733636 DEBUG (ScheduledTrigger-11897-thread-3) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_lost_trigger, node_added_trigger]
   [junit4]   2> 4733638 DEBUG (AutoscalingActionExecutor-11898-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"1105f676dec693T84uj8qni1ikeu31q71tqcgaoo",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":4791630718486163,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[4791630718486163],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":4791632177137563,
   [junit4]   2>     "nodeNames":["127.0.0.1:10018_solr"]}}
   [junit4]   2> 4733639 DEBUG (AutoscalingActionExecutor-11898-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 4733639 DEBUG (AutoscalingActionExecutor-11898-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger after 100ms
   [junit4]   2> 4733639 INFO  (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeMarkersRegistration
   [junit4]   2> 4733639 DEBUG (AutoscalingActionExecutor-11898-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
   [junit4]   2> 4733639 DEBUG (AutoscalingActionExecutor-11898-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 70 ms for event id=1105f676dec693T84uj8qni1ikeu31q71tqcgaoo
   [junit4]   2> 4733639 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=16
   [junit4]   2> 4733640 DEBUG (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 16
   [junit4]   2> 4733640 INFO  (TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes: 2
   [junit4]   2> ## Empty nodes: 1
   [junit4]   2> ## Dead nodes: 2
   [junit4]   2> ## 127.0.0.1:10017_solr
   [junit4]   2> ## 127.0.0.1:10018_solr
   [junit4]   2> ## Collections:
   [junit4]   2> ##  * .system
   [junit4]   2> ##    shardsTotal 1
   [junit4]   2> ##    shardsState {active=1}
   [junit4]   2> ##      shardsWithoutLeader 0
   [junit4]   2> ##    totalReplicas 2
   [junit4]   2> ##      activeReplicas 2
   [junit4]   2> ##      inactiveReplicas 0
   [junit4]   2> ##    totalActiveDocs 4
   [junit4]   2> ##      bufferedDocs 0
   [junit4]   2> ##      maxActiveSliceDocs 4
   [junit4]   2> ##      minActiveSliceDocs 4
   [junit4]   2> ##      avgActiveSliceDocs 4
   [junit4]   2> ##    totalInactiveDocs 0
   [junit4]   2> ##      maxInactiveSliceDocs 0
   [junit4]   2> ##      minInactiveSliceDocs 0
   [junit4]   2> ##      avgInactiveSliceDocs NaN
   [junit4]   2> ##    totalActiveBytes 12,240
   [junit4]   2> ##      maxActiveSliceBytes 12,240
   [junit4]   2> ##      minActiveSliceBytes 12,240
   [junit4]   2> ##      avgActiveSliceBytes 12,240
   [junit4]   2> ##    totalInactiveBytes 0
   [junit4]   2> ##      maxInactiveSliceBytes 0
   [junit4]   2> ##      minInactiveSliceBytes 0
   [junit4]   2> ##      avgInactiveSliceBytes NaN
   [junit4]   2> ##    totalActiveDeletedDocs 0
   [junit4]   2> ## Max replicas per node: 1
   [junit4]   2> ## Min replicas per node: 0
   [junit4]   2> ## Total replicas: 1
   [junit4]   2> ## * .system 1
   [junit4]   2> ## - active           1
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ## - autoscaling        3
   [junit4]   2> ## - update             4
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: node_added_trigger
   [junit4]   2> ## - STARTED         1
   [junit4]   2> ## - SUCCEEDED       1
   [junit4]   2> ## * Trigger: node_added_trigger1
   [junit4]   2> ## - STARTED         2
   [junit4]   2> ## - SUCCEEDED       1
   [junit4]   2> ## * Trigger: node_lost_trigger
   [junit4]   2> ## - STARTED         1
   [junit4]   2> ## - SUCCEEDED       1
   [junit4]   2>
   [junit4]   2> 4733675 INFO  (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[BB5D3569FE6D602A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTriggerRestoreState
   [junit4]   2> 4733675 DEBUG (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
   [junit4]   2> 4733675 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
   [junit4]   2> 4733675 DEBUG (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10018_solr
   [junit4]   2> 4733676 DEBUG (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10017_solr
   [junit4]   2> 4733676 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 4733676 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 4733676 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
   [junit4]   2> 4733678 INFO  (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 4733679 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 4733681 DEBUG (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 4733681 DEBUG (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 4733681 DEBUG (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 4733681 DEBUG (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 4733681 DEBUG (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 4733681 DEBUG (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[BB5D3569FE6D602A]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 4733683 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 4733683 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 11
   [junit4]   2> 4733683 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion -1
   [junit4]   2> 4733683 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
   [junit4]   2> 4733684 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10019_solr]
   [junit4]   2> 4733684 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 4733684 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion 11
   [junit4]   2> 4733684 DEBUG (ScheduledTrigger-11900-thread-1) [    ] o.a.s.c.a.NodeLostTrigg

[...truncated too long message...]

fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/null1090641855
     [copy] Copying 241 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/null1090641855
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/null1090641855

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 48ms :: artifacts dl 4ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 376 minutes 50 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-NightlyTests-7.x - Build # 38 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-7.x/38/

4 tests failed.
FAILED:  org.apache.solr.cloud.BasicDistributedZk2Test.test

Error Message:
shard1 is not consistent.  Got 36 from http://127.0.0.1:33915/z_/q/collection1_shard1_replica_n1 (previous client) and got 33 from http://127.0.0.1:38946/z_/q/collection1_shard1_replica_n6

Stack Trace:
java.lang.AssertionError: shard1 is not consistent.  Got 36 from http://127.0.0.1:33915/z_/q/collection1_shard1_replica_n1 (previous client) and got 33 from http://127.0.0.1:38946/z_/q/collection1_shard1_replica_n6
        at __randomizedtesting.SeedInfo.seed([EA66F8357AB315C2:6232C7EFD44F783A]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1423)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1402)
        at org.apache.solr.cloud.BasicDistributedZk2Test.brindDownShardIndexSomeDocsAndRecover(BasicDistributedZk2Test.java:373)
        at org.apache.solr.cloud.BasicDistributedZk2Test.test(BasicDistributedZk2Test.java:108)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1063)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1035)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest

Error Message:
ObjectTracker found 2 object(s) that were not released!!! [SolrZkClient, ZkStateReader] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:203)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:126)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:116)  at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:306)  at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160)  at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:399)  at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)  at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)  at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)  at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)  at org.apache.solr.client.solrj.impl.SolrClientCloudManager.request(SolrClientCloudManager.java:115)  at org.apache.solr.cloud.autoscaling.ExecutePlanAction.waitForTaskToFinish(ExecutePlanAction.java:132)  at org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:85)  at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:309)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.ZkStateReader  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:328)  at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160)  at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:399)  at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)  at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)  at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)  at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)  at org.apache.solr.client.solrj.impl.SolrClientCloudManager.request(SolrClientCloudManager.java:115)  at org.apache.solr.cloud.autoscaling.ExecutePlanAction.waitForTaskToFinish(ExecutePlanAction.java:132)  at org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:85)  at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:309)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 2 object(s) that were not released!!! [SolrZkClient, ZkStateReader]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:203)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:126)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:116)
        at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:306)
        at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:399)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.client.solrj.impl.SolrClientCloudManager.request(SolrClientCloudManager.java:115)
        at org.apache.solr.cloud.autoscaling.ExecutePlanAction.waitForTaskToFinish(ExecutePlanAction.java:132)
        at org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:85)
        at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:309)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.ZkStateReader
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:328)
        at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:399)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.client.solrj.impl.SolrClientCloudManager.request(SolrClientCloudManager.java:115)
        at org.apache.solr.cloud.autoscaling.ExecutePlanAction.waitForTaskToFinish(ExecutePlanAction.java:132)
        at org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:85)
        at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:309)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


        at __randomizedtesting.SeedInfo.seed([EA66F8357AB315C2]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:317)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest

Error Message:
6 threads leaked from SUITE scope at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest:     1) Thread[id=107553, name=zkCallback-7109-thread-3, state=TIMED_WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=107279, name=zkConnectionManagerCallback-7110-thread-1, state=WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=107278, name=AutoscalingActionExecutor-6888-thread-1-EventThread, state=WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)    4) Thread[id=107312, name=zkCallback-7109-thread-1, state=TIMED_WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=107277, name=AutoscalingActionExecutor-6888-thread-1-SendThread(127.0.0.1:44026), state=TIMED_WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)    6) Thread[id=107552, name=zkCallback-7109-thread-2, state=TIMED_WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 6 threads leaked from SUITE scope at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest:
   1) Thread[id=107553, name=zkCallback-7109-thread-3, state=TIMED_WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=107279, name=zkConnectionManagerCallback-7110-thread-1, state=WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=107278, name=AutoscalingActionExecutor-6888-thread-1-EventThread, state=WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
   4) Thread[id=107312, name=zkCallback-7109-thread-1, state=TIMED_WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=107277, name=AutoscalingActionExecutor-6888-thread-1-SendThread(127.0.0.1:44026), state=TIMED_WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
   6) Thread[id=107552, name=zkCallback-7109-thread-2, state=TIMED_WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([EA66F8357AB315C2]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=107553, name=zkCallback-7109-thread-3, state=TIMED_WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=107279, name=zkConnectionManagerCallback-7110-thread-1, state=WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=107312, name=zkCallback-7109-thread-1, state=TIMED_WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=107277, name=AutoscalingActionExecutor-6888-thread-1-SendThread(127.0.0.1:44026), state=TIMED_WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)    5) Thread[id=107552, name=zkCallback-7109-thread-2, state=TIMED_WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=107553, name=zkCallback-7109-thread-3, state=TIMED_WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=107279, name=zkConnectionManagerCallback-7110-thread-1, state=WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=107312, name=zkCallback-7109-thread-1, state=TIMED_WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=107277, name=AutoscalingActionExecutor-6888-thread-1-SendThread(127.0.0.1:44026), state=TIMED_WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)
   5) Thread[id=107552, name=zkCallback-7109-thread-2, state=TIMED_WAITING, group=TGRP-SharedFSAutoReplicaFailoverTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([EA66F8357AB315C2]:0)




Build Log:
[...truncated 13734 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/init-core-data-001
   [junit4]   2> 1230084 WARN  (SUITE-BasicDistributedZk2Test-seed#[EA66F8357AB315C2]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 1230085 INFO  (SUITE-BasicDistributedZk2Test-seed#[EA66F8357AB315C2]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1230102 INFO  (SUITE-BasicDistributedZk2Test-seed#[EA66F8357AB315C2]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 1230102 INFO  (SUITE-BasicDistributedZk2Test-seed#[EA66F8357AB315C2]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1230102 INFO  (SUITE-BasicDistributedZk2Test-seed#[EA66F8357AB315C2]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /z_/q
   [junit4]   2> 1230105 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1230114 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1230114 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1230160 ERROR (ZkTestServer Run Thread) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1230214 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.ZkTestServer start zk server on port:38957
   [junit4]   2> 1230214 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:38957
   [junit4]   2> 1230214 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 38957
   [junit4]   2> 1230260 INFO  (zkConnectionManagerCallback-4100-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1230288 INFO  (zkConnectionManagerCallback-4102-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1230289 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1230290 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1230291 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1230292 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1230292 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1230293 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1230294 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1230294 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1230295 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1230296 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1230297 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1230297 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 1230545 WARN  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1230545 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1230545 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1230545 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1230632 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1230632 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1230632 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1230633 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@16a51dd7{/z_/q,null,AVAILABLE}
   [junit4]   2> 1230641 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@34eadd02{HTTP/1.1,[http/1.1]}{127.0.0.1:46421}
   [junit4]   2> 1230641 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.e.j.s.Server Started @1230694ms
   [junit4]   2> 1230641 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/tempDir-001/control/data, hostContext=/z_/q, hostPort=46421, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/control-001/cores}
   [junit4]   2> 1230641 ERROR (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1230641 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1230641 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 1230641 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1230641 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1230641 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-12-06T00:35:39.724Z
   [junit4]   2> 1230680 INFO  (zkConnectionManagerCallback-4104-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1230684 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1230684 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/control-001/solr.xml
   [junit4]   2> 1230687 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1230687 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1230689 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3c8b2fa1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1231441 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38957/solr
   [junit4]   2> 1231461 INFO  (zkConnectionManagerCallback-4108-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1231492 INFO  (zkConnectionManagerCallback-4110-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1231566 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46421_z_%2Fq
   [junit4]   2> 1231567 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.c.Overseer Overseer (id=73787347299139588-127.0.0.1:46421_z_%2Fq-n_0000000000) starting
   [junit4]   2> 1231632 INFO  (zkConnectionManagerCallback-4117-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1231634 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38957/solr ready
   [junit4]   2> 1231655 INFO  (OverseerStateUpdate-73787347299139588-127.0.0.1:46421_z_%2Fq-n_0000000000) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:46421_z_%2Fq
   [junit4]   2> 1231668 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46421_z_%2Fq
   [junit4]   2> 1231680 INFO  (zkCallback-4116-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1231680 INFO  (zkCallback-4109-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1231707 INFO  (OverseerAutoScalingTriggerThread-73787347299139588-127.0.0.1:46421_z_%2Fq-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:46421_z_%2Fq]
   [junit4]   2> 1231712 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1231736 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c8b2fa1
   [junit4]   2> 1231740 INFO  (ScheduledTrigger-2016-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1
   [junit4]   2> 1231751 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c8b2fa1
   [junit4]   2> 1231751 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c8b2fa1
   [junit4]   2> 1231753 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/control-001/cores
   [junit4]   2> 1231844 INFO  (zkConnectionManagerCallback-4123-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1231844 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1231845 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38957/solr ready
   [junit4]   2> 1231856 INFO  (qtp710659866-9948) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:46421_z_%252Fq&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1231892 INFO  (OverseerThreadFactory-2018-thread-1-processing-n:127.0.0.1:46421_z_%2Fq) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 1232014 INFO  (qtp710659866-9950) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1232015 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1232020 INFO  (qtp710659866-9947) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1232021 INFO  (qtp710659866-9949) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1232022 INFO  (qtp710659866-9950) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1232032 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1232032 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1232742 INFO  (ScheduledTrigger-2016-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1
   [junit4]   2> 1233060 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 1233115 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 1233455 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1233507 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 1233508 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c8b2fa1
   [junit4]   2> 1233508 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 1233511 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=825790686, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1233525 WARN  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1233688 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1233688 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1233704 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1233704 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1233716 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=927521368, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1233717 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6de50424[control_collection_shard1_replica_n1] main]
   [junit4]   2> 1233718 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1233718 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1233719 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1233719 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1619060633425149952
   [junit4]   2> 1233722 INFO  (searcherExecutor-2023-thread-1-processing-n:127.0.0.1:46421_z_%2Fq x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@6de50424[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1233738 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 1233738 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/control_collection/leaders/shard1
   [junit4]   2> 1233741 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1233741 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1233741 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46421/z_/q/control_collection_shard1_replica_n1/
   [junit4]   2> 1233741 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1233741 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:46421/z_/q/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 1233741 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1233741 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/73787347299139588-core_node2-n_0000000000
   [junit4]   2> 1233742 INFO  (ScheduledTrigger-2016-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1
   [junit4]   2> 1233743 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46421/z_/q/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 1233845 INFO  (zkCallback-4109-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1233846 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1233847 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1815
   [junit4]   2> 1233859 INFO  (zkCallback-4109-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1233869 INFO  (qtp710659866-9948) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 1233900 INFO  (OverseerCollectionConfigSetProcessor-73787347299139588-127.0.0.1:46421_z_%2Fq-n_0000000000) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1233960 INFO  (zkCallback-4109-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1233960 INFO  (zkCallback-4109-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1233969 INFO  (zkCallback-4109-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1233969 INFO  (qtp710659866-9948) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:46421_z_%252Fq&wt=javabin&version=2} status=0 QTime=2112
   [junit4]   2> 1234044 INFO  (zkConnectionManagerCallback-4128-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1234045 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1234045 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38957/solr ready
   [junit4]   2> 1234045 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1234067 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1234084 INFO  (OverseerThreadFactory-2018-thread-2-processing-n:127.0.0.1:46421_z_%2Fq) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 1234289 WARN  (OverseerThreadFactory-2018-thread-2-processing-n:127.0.0.1:46421_z_%2Fq) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 1234291 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 1234292 INFO  (qtp710659866-9951) [n:127.0.0.1:46421_z_%2Fq    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=225
   [junit4]   2> 1234308 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating jetty instances pullReplicaCount=0 numOtherReplicas=4
   [junit4]   2> 1234595 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/shard-1-001 of type NRT
   [junit4]   2> 1234612 WARN  (closeThreadPool-4129-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1234614 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1234614 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1234614 INFO  (closeThreadPool-4129-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1234615 INFO  (closeThreadPool-4129-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1234615 INFO  (closeThreadPool-4129-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1234615 INFO  (closeThreadPool-4129-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1234615 INFO  (closeThreadPool-4129-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2bee7d43{/z_/q,null,AVAILABLE}
   [junit4]   2> 1234640 INFO  (closeThreadPool-4129-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@45c96ff8{HTTP/1.1,[http/1.1]}{127.0.0.1:36839}
   [junit4]   2> 1234640 INFO  (closeThreadPool-4129-thread-1) [    ] o.e.j.s.Server Started @1234693ms
   [junit4]   2> 1234640 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/z_/q, hostPort=36839, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/shard-1-001/cores}
   [junit4]   2> 1234640 ERROR (closeThreadPool-4129-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1234640 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1234640 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 1234640 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1234640 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1234640 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-12-06T00:35:43.723Z
   [junit4]   2> 1234656 INFO  (zkConnectionManagerCallback-4131-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1234656 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1234656 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/shard-1-001/solr.xml
   [junit4]   2> 1234660 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1234660 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1234662 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3c8b2fa1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1234742 INFO  (ScheduledTrigger-2016-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1
   [junit4]   2> 1234881 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/shard-2-001 of type NRT
   [junit4]   2> 1234892 WARN  (closeThreadPool-4129-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1234893 INFO  (closeThreadPool-4129-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1234893 INFO  (closeThreadPool-4129-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1234893 INFO  (closeThreadPool-4129-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1235004 INFO  (closeThreadPool-4129-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1235004 INFO  (closeThreadPool-4129-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1235004 INFO  (closeThreadPool-4129-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1235012 INFO  (closeThreadPool-4129-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@153225aa{/z_/q,null,AVAILABLE}
   [junit4]   2> 1235014 INFO  (closeThreadPool-4129-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3ad5adb3{HTTP/1.1,[http/1.1]}{127.0.0.1:33915}
   [junit4]   2> 1235014 INFO  (closeThreadPool-4129-thread-2) [    ] o.e.j.s.Server Started @1235067ms
   [junit4]   2> 1235014 INFO  (closeThreadPool-4129-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/z_/q, hostPort=33915, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/shard-2-001/cores}
   [junit4]   2> 1235014 ERROR (closeThreadPool-4129-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1235014 INFO  (closeThreadPool-4129-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1235014 INFO  (closeThreadPool-4129-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 1235014 INFO  (closeThreadPool-4129-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1235014 INFO  (closeThreadPool-4129-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1235014 INFO  (closeThreadPool-4129-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-12-06T00:35:44.097Z
   [junit4]   2> 1235064 INFO  (zkConnectionManagerCallback-4133-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1235068 INFO  (closeThreadPool-4129-thread-2) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1235068 INFO  (closeThreadPool-4129-thread-2) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/shard-2-001/solr.xml
   [junit4]   2> 1235072 INFO  (closeThreadPool-4129-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1235072 INFO  (closeThreadPool-4129-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1235073 INFO  (closeThreadPool-4129-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3c8b2fa1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1235136 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38957/solr
   [junit4]   2> 1235176 INFO  (zkConnectionManagerCallback-4137-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1235204 INFO  (zkConnectionManagerCallback-4139-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1235207 INFO  (closeThreadPool-4129-thread-1) [n:127.0.0.1:36839_z_%2Fq    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1235210 INFO  (closeThreadPool-4129-thread-1) [n:127.0.0.1:36839_z_%2Fq    ] o.a.s.c.ZkController Publish node=127.0.0.1:36839_z_%2Fq as DOWN
   [junit4]   2> 1235210 INFO  (closeThreadPool-4129-thread-1) [n:127.0.0.1:36839_z_%2Fq    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1235210 INFO  (closeThreadPool-4129-thread-1) [n:127.0.0.1:36839_z_%2Fq    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36839_z_%2Fq
   [junit4]   2> 1235212 INFO  (zkCallback-4116-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1235212 INFO  (zkCallback-4127-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1235212 INFO  (zkCallback-4109-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1235232 INFO  (zkCallback-4138-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1235304 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/shard-3-001 of type NRT
   [junit4]   2> 1235308 INFO  (zkConnectionManagerCallback-4146-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1235309 INFO  (closeThreadPool-4129-thread-1) [n:127.0.0.1:36839_z_%2Fq    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1235309 INFO  (closeThreadPool-4129-thread-1) [n:127.0.0.1:36839_z_%2Fq    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38957/solr ready
   [junit4]   2> 1235325 WARN  (closeThreadPool-4129-thread-3) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1235326 INFO  (closeThreadPool-4129-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1235326 INFO  (closeThreadPool-4129-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1235326 INFO  (closeThreadPool-4129-thread-3) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1235340 INFO  (closeThreadPool-4129-thread-1) [n:127.0.0.1:36839_z_%2Fq    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1235361 INFO  (closeThreadPool-4129-thread-1) [n:127.0.0.1:36839_z_%2Fq    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c8b2fa1
   [junit4]   2> 1235391 INFO  (closeThreadPool-4129-thread-1) [n:127.0.0.1:36839_z_%2Fq    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c8b2fa1
   [junit4]   2> 1235391 INFO  (closeThreadPool-4129-thread-1) [n:127.0.0.1:36839_z_%2Fq    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c8b2fa1
   [junit4]   2> 1235413 INFO  (closeThreadPool-4129-thread-1) [n:127.0.0.1:36839_z_%2Fq    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/shard-1-001/cores
   [junit4]   2> 1235416 INFO  (closeThreadPool-4129-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1235416 INFO  (closeThreadPool-4129-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1235416 INFO  (closeThreadPool-4129-thread-3) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1235440 INFO  (closeThreadPool-4129-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1cb498e1{/z_/q,null,AVAILABLE}
   [junit4]   2> 1235472 INFO  (closeThreadPool-4129-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@51362999{HTTP/1.1,[http/1.1]}{127.0.0.1:40661}
   [junit4]   2> 1235472 INFO  (closeThreadPool-4129-thread-3) [    ] o.e.j.s.Server Started @1235525ms
   [junit4]   2> 1235472 INFO  (closeThreadPool-4129-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/z_/q, hostPort=40661, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/shard-3-001/cores}
   [junit4]   2> 1235473 ERROR (closeThreadPool-4129-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1235473 INFO  (closeThreadPool-4129-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1235473 INFO  (closeThreadPool-4129-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 1235473 INFO  (closeThreadPool-4129-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1235473 INFO  (closeThreadPool-4129-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1235473 INFO  (closeThreadPool-4129-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-12-06T00:35:44.556Z
   [junit4]   2> 1235514 INFO  (zkConnectionManagerCallback-4149-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1235515 INFO  (closeThreadPool-4129-thread-3) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1235515 INFO  (closeThreadPool-4129-thread-3) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/shard-3-001/solr.xml
   [junit4]   2> 1235518 INFO  (closeThreadPool-4129-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1235518 INFO  (closeThreadPool-4129-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1235520 INFO  (closeThreadPool-4129-thread-3) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3c8b2fa1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1235594 INFO  (closeThreadPool-4129-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38957/solr
   [junit4]   2> 1235664 INFO  (zkConnectionManagerCallback-4153-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1235682 INFO  (TEST-BasicDistributedZk2Test.test-seed#[EA66F8357AB315C2]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 4 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/shard-4-001 of type NRT
   [junit4]   2> 1235682 WARN  (closeThreadPool-4129-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1235683 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1235683 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1235683 INFO  (closeThreadPool-4129-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1235742 INFO  (ScheduledTrigger-2016-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 1
   [junit4]   2> 1235752 INFO  (closeThreadPool-4129-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1235752 INFO  (closeThreadPool-4129-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1235752 INFO  (closeThreadPool-4129-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1235760 INFO  (closeThreadPool-4129-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6e54578{/z_/q,null,AVAILABLE}
   [junit4]   2> 1235777 INFO  (closeThreadPool-4129-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@692eda7d{HTTP/1.1,[http/1.1]}{127.0.0.1:38946}
   [junit4]   2> 1235777 INFO  (closeThreadPool-4129-thread-1) [    ] o.e.j.s.Server Started @1235830ms
   [junit4]   2> 1235777 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/tempDir-001/jetty4, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/z_/q, hostPort=38946, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/shard-4-001/cores}
   [junit4]   2> 1235778 ERROR (closeThreadPool-4129-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1235778 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1235778 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 1235778 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1235778 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1235778 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-12-06T00:35:44.861Z
   [junit4]   2> 1235792 INFO  (zkConnectionManagerCallback-4155-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1235796 INFO  (closeThreadPool-4129-thread-2) [n:127.0.0.1:33915_z_%2Fq    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1235805 INFO  (closeThreadPool-4129-thread-2) [n:127.0.0.1:33915_z_%2Fq    ] o.a.s.c.ZkController Publish node=127.0.0.1:33915_z_%2Fq as DOWN
   [junit4]   2> 1235806 INFO  (closeThreadPool-4129-thread-2) [n:127.0.0.1:33915_z_%2Fq    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1235806 INFO  (closeThreadPool-4129-thread-2) [n:127.0.0.1:33915_z_%2Fq    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33915_z_%2Fq
   [junit4]   2> 1235807 INFO  (zkCallback-4116-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1235807 INFO  (zkCallback-4127-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1235807 INFO  (zkCallback-4109-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1235807 INFO  (zkCallback-4138-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1235808 INFO  (zkConnectionManagerCallback-4157-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1235816 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x106253390af0010, likely client has closed socket
   [junit4]   2> 1235842 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1235842 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZk2Test_EA66F8357AB315C2-001/shard-4-001/solr.xml
   [junit4]   2> 1235843 INFO  (zkCallback-4145-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1235846 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1235846 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1235847 INFO  (closeThreadPool-4129-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3c8b2fa1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1235852 INFO  (zkCallback-4154-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1235927 INFO  (zkConnectionManagerCallback-4164-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1235932 INFO  (closeThreadPool-4129-thread-2) [n:127.0.0.1:33915_z_%2Fq    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1235938 INFO  (closeThreadPool-4129-thread-2) [n:127.0.0.1:33915_z_%2Fq    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38957/solr ready
   [junit4]   2> 1235960 INFO  (closeThreadPool-4129-thread-2) [n:127.0.0.1:33915_z_%2Fq    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1236039 INFO  (closeThreadPool-4129-thread-2) [n:127.0.0.1:33915_z_%2Fq    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c8b2fa1
   [junit4]   2> 1236064 INFO  (closeThreadPool-4129-thread-2) [n:127.0.0.1:33915_z_%2Fq    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c8b2fa1
   [junit4]   2> 1236064 INFO  (closeThreadPool-4129-thread-2) [n:127.0.0.1:33915_z_%2Fq    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c8b2fa1
   [junit4]   2> 1236066 INFO  (closeThreadPool-4129-thread-2) [n:127.0.0.1:33915_z_%2Fq    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Ba

[...truncated too long message...]

15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/api-all-1.0.0-M20.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/bcprov-jdk15on-1.60.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/byte-buddy-1.9.3.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/ehcache-core-2.4.4.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/hadoop-common-2.7.4-tests.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/hadoop-hdfs-2.7.4-tests.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/hadoop-minikdc-2.7.4.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/jersey-core-1.9.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/jersey-server-1.9.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/jetty-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/jetty-sslengine-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/jetty-util-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/mina-core-2.0.0-M5.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/mockito-core-2.23.4.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/netty-all-4.0.36.Final.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/test-lib/objenesis-2.6.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/build/analysis/icu/lucene-analyzers-icu-7.7.0-SNAPSHOT.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/contrib/solr-analysis-extras/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/build/queryparser/classes/test:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/build/backward-codecs/classes/test:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/contrib/analysis-extras/lib/icu4j-62.1.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/usr/local/asfpackages/java/jdk1.8.0_191/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.6.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J2-20181206_001508_9843043003268745629560.events @/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J2-20181206_001508_9847033517753969683158.suites -stdin
   [junit4] ERROR: JVM J2 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1563)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:123)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:997)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:994)
   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4] at java.lang.Thread.run(Thread.java:748)

BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/build.xml:661: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/build.xml:577: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/build.xml:59: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build.xml:267: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/common-build.xml:558: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/common-build.xml:1572: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/common-build.xml:1099: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 543 minutes 31 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[Fast Archiver] No artifacts from Lucene-Solr-BadApples-NightlyTests-7.x #37 to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-NightlyTests-7.x - Build # 39 - Still Failing

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-7.x/39/

No tests ran.

Build Log:
[...truncated 6 lines...]
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git://git.apache.org/lucene-solr.git
        at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
        at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
        at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
        at hudson.scm.SCM.checkout(SCM.java:504)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
        at hudson.model.Run.execute(Run.java:1794)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url git://git.apache.org/lucene-solr.git" returned status code 4:
stdout:
stderr: error: failed to write new configuration file /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/.git/config.lock

        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2002)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1970)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1966)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1597)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1609)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1243)
        at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
        at sun.reflect.GeneratedMethodAccessor595.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
        at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
        at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
                at com.sun.proxy.$Proxy117.setRemoteUrl(Unknown Source)
                at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
                at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:876)
                at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
                at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
                at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
                at hudson.scm.SCM.checkout(SCM.java:504)
                at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
                at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
                at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
                at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
                at hudson.model.Run.execute(Run.java:1794)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git://git.apache.org/lucene-solr.git # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git://git.apache.org/lucene-solr.git
        at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
        at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
        at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
        at hudson.scm.SCM.checkout(SCM.java:504)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
        at hudson.model.Run.execute(Run.java:1794)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url git://git.apache.org/lucene-solr.git" returned status code 4:
stdout:
stderr: error: failed to write new configuration file /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/.git/config.lock

        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2002)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1970)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1966)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1597)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1609)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1243)
        at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
        at sun.reflect.GeneratedMethodAccessor595.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
        at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
        at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
                at com.sun.proxy.$Proxy117.setRemoteUrl(Unknown Source)
                at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
                at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:876)
                at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
                at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
                at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
                at hudson.scm.SCM.checkout(SCM.java:504)
                at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
                at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
                at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
                at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
                at hudson.model.Run.execute(Run.java:1794)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git://git.apache.org/lucene-solr.git # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git://git.apache.org/lucene-solr.git
        at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
        at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
        at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
        at hudson.scm.SCM.checkout(SCM.java:504)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
        at hudson.model.Run.execute(Run.java:1794)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url git://git.apache.org/lucene-solr.git" returned status code 4:
stdout:
stderr: error: failed to write new configuration file /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/.git/config.lock

        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2002)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1970)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1966)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1597)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1609)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1243)
        at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
        at sun.reflect.GeneratedMethodAccessor595.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
        at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
        at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
                at com.sun.proxy.$Proxy117.setRemoteUrl(Unknown Source)
                at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
                at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:876)
                at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
                at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
                at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
                at hudson.scm.SCM.checkout(SCM.java:504)
                at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
                at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
                at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
                at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
                at hudson.model.Run.execute(Run.java:1794)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
Archiving artifacts
[Fast Archiver] No artifacts from Lucene-Solr-BadApples-NightlyTests-7.x #37 to compare, so performing full copy of artifacts
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run?
For example, /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/lucene/build/analysis/common/test/TEST-org.apache.lucene.analysis.ar.TestArabicAnalyzer.xml is 5 days 14 hr old

Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]